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I.  XMTRODUCTZON 


PURPOSE 

The  purpose  of  this  research  project  was  to  identify  and 
assess  the  quality  assurance  methods  used  by  providers  of  financial 
management  education  and  training  courses  and  programs  offered 
throughout  the  DoD.  On  July  3,  1991,  the  DoD  Comptroller  issued  a 
memorandum  entitled  "Management  Plan  for  the  Review  of  Financial 
Management  Education  and  Training".  [Ref.  7:]  This  memorandum 
addressed  a  broad  spectrum  of  financial  management  issues,  and  also 
spoke  to  the  issue  of  (quality  control.  Under  the  heading  of  Needs 
Assessment,  curriculum  configuration  management  and  quality  control 
was  addressed  as  follows: 

Financial  management  course  offerings  fall  generally  into  two 
categories:  core  courses  that  deal  in  basic  concepts  and 

principles  that  are  independent  of  specific  application  ...; 
and  service  unique  applications  that  train  personnel  to 
perform  relatively  narrow  functions  or  apply  basic  principles 
in  situations  peculiar  to  a  particular  Service. 

In  the  case  of  core  courses  there  is  a  need  to  ensure  that  the 

course  content  covers  the  subject  matter  at  an  appropriate  level  of 

comprehensiveness  and  sophistication.  A  second,  related  function 

is  that  of  ensuring  that  what  is  taught  is  taught  well.  Whether 

analyzing  new  proposals  or  evaluating  ongoing  programs  of 

instruction,  assessments  of  teaching  effectiveness  are  needed. 

[Ref.  7:] 

A  survey  was  administered  to  collect  detailed  information  on 
methods  employed  by  the  various  agencies  and  departments  to  ensure 
the  effectiveness  of  their  courses  and  programs.  The  providers  of 
financial  management  education  and  training  courses  and  programs 
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were  asked  to  complete  a  detailed  questionnaire  designed  to  assess 
their  programmatic  offerings.  All  providers  of  financial  management 
education  and  training  In  DOD  were  surveyed. 

All  military  departments  and  defense  agencies  such  as  the 
Defense  Logistics  Agency  and  the  Defense  Finance  and  Accounting 
Service  are  Included  In  the  study.  In-house  as  well  as  outside 
contracted  providers  were  Included,  as  were  both  civilian  and 
military  financial  management  education  and  training  programs. 
However,  financial  management  education  and  training  programs  based 
on  the  "correspondence"  method  were  excluded  from  this  research. 

In  summary,  this  report  attempts  to  answer  the  question  of 
what  methods  are  employed  to  assure  the  quality  of  financial 
management  courses  and  programs  within  DoD. 

METHODOLOGY 

The  methodology  for  conducting  this  research  involved  four 
distinct  steps.  Initially,  the  research  concentrated  on  a  thorough 
review  of  current  literature,  instructions,  directives,  reference 
materials  and  guidance  dealing  with  financial  management  education 
and  training.  Information  specifically  concerned  with  financial 
management  training  course  availability,  course  content,  and 
targeted  populations  was  reviewed.  Applicable  instructions  and 
guidance  for  financial  management  personnel  were  also  reviewed. 
Finally,  the  literature  on  quality  assessment  and  assessment 
techniques  was  reviewed. 

Secondly,  the  development  of  a  comprehensive  survey  instrument 
was  undertaken.  To  help  ensure  a  relevant  and  comprehensive  survey 
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questionnaire  was  prepared,  telephone  interviews  were  conducted 
with  administrators  and  professionals  employed  by  providers  of 
financial  management  education  and  training  within  DoD.  To  verify 
the  contents  and  organization  of  the  questionnaire,  a  pilot  test 
was  conducted  with  two  providers  of  financial  management  education 
and  training.  The  questionnaire  was  then  distributed  to  all 
institutions  within  DoD  providing  financial  management  education 
and  training.  These  institutions  were  identified  by  using  the  DoD 
Training  and  Performance  Data  Center  listing  of  financial 
management  education  and  training  providers.  The  appendix  lists 
agencies  that  responded  to  the  survey.  Finally,  the  survey 
responses  were  tabulated  and  analyzed,  and  in  some  cases  telephone 
follow-up  was  conducted  to  clarify  data  from  the  responses. 

SCOPE 

A  study  conducted  in  1990  by  the  Defense  Manpower  Data  Center 
(DMDC)  details  the  number  of  civilian  and  service  personnel 
involved  in  financial/resource  management.  While  the  aggregate 
numbers  presented  in  Table  I  below  have  been  questioned  by  the 
military  departments  [Ref.  3;  p.l89]  within  DoD  (e.g.,  the  total 
acknowledged  by  these  components  is  approximately  20,000  less  than 
estimated  by  DMDC) ,  the  numbers  generally  represent  the  total 
number  of  personnel  engaged  DoD-wide  in  the  financial  or  resource 
management  field.  Table  I  details  the  findings  of  the  DMDC 
study. [Ref.  3:  pp.  187-188] 
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TABLE  I.  FIHANCIAL/RESOORCB  MAKAGEMENT  EMPLOYEES  OF  DoD 


CIVILIAN  {  MILITARY 


AGENCY 

PRPF/M>MIN 

TECHNICAL 

gyPTQBR 

ENLISTED 

TOTAL 

ARMY 

27,105 

13,014 

2,746 

3,942 

46,807 

13,945 

5,647 

26,519 

NAVY 

15,513 

9,051 

688 

2,872 

28,124 

MC 

921 

1,194 

304 

1,314 

3,733 

DLA 

2,902 

2,671 

5,573 

OTHER 

8,124 

292 

8,416 

TOTAL 

68,510 

31,869 

13,504 

119,172 

FINANCIAL  MANAGEMENT  EDUCATION  AND  TRAINING 


The  task  of  training  and  educating  the  100,000  or  more 
personnel  involved  in  financial  management  is  a  major  enterprize. 
The  training  and  education  provided  ranges  from  basic  introductory 
courses  in  payroll,  accounting  and  disbursing  for  junior  enlisted 
personnel  to  graduate  level  Master's  programs  in  financial 
management  for  military  officers  and  mid-grade  civilian  employees. 

A  total  of  29  agencies  and  departments  provide  financial 
management  and  training.  This  number  does  not  include  the 
correspondence  courses  offered  by  the  various  military  departments 
nor  does  it  account  for  education  obtained  by  military  and  civilian 
personnel  outside  of  the  structured  programs  recognized  in  this 
research.  The  diversity  and  number  of  courses  offered  are 
documented  in  Table  II.  [Ref.  3:  p.  127] 
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ENMICIAL  MAMAGEMEMT  COPRSES  BY  PROVIDER 


Functional  area: 

Army 

Navy 

Air  Force 

DLA 

Total 

BMdqgt 

9 

2 

— 

13 

Financ? 

37 

23 

19 

12 

91 

Analygjs 

27 

1 

19 

— 

47 

Resource  Momt 

27 

1 

— 

— 

28 

Totals 

100 

27 

40 

12 

179 

As  Table  II  demonstrates. 

the  31 

institutions 

offer 

179  different 

courses  of  instruction.  While  firm  data  are  not  available,  it  is 
estimated  that  at  least  10  percent  of  the  financial  management 
workforce,  approximately,  10,000  personnel,  attend  financial 
management  education  and  training  programs  and  courses  of  some  kind 
each  year.  [Ref.  3] 

II.  SURVEY  RESPONSES 

This  section  summarizes  the  findings  obtained  from  the  survey 
responses  returned  by  financial  management  education  and  training 
providers,  first,  by  general  characteristics  and  then  by  question. 

A.  General  Characteristics  of  Financial  Management  Education 
and  Training  courses  and  Programs 

Twenty-one  of  the  thirty-one  institutions  surveyed  responded 
within  the  time  requirements  specified  in  the  survey  instrument.  Of 
the  21  responses  to  the  survey  19  agencies  provided  detailed 
responses  and  two  agencies  replied  that  they  did  not  conduct 
financial  management  education  and  training. 

The  survey  questionnaire  was  designed  to  gather  basic  data  on 
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the  characteristics  of  financial  management  education  and  training 
providers.  Questions  one  through  ten  provided  the  respondents  with 
the  opportunity  to  describe  their  courses  or  programs  in  terms  of 
target  audience;  number  and  length  of  those  courses  and  programs; 
enrollment;  average  class  size; and  the  level  of  instruction  (  e. 
g . ,  degree  granting) .  Several  other  questions  also  gathered 
general  program  information  on  these  providers.  For  example, 
question  32  asked  if  the  financial  management  program  was  supported 
by  a  library  containing  extensive  DoD  financial  management  related 
materials.  The  following  analysis  details  the  general 
characteristics  of  financial  management  education  and  training 
providers  responding  to  the  survey  questionnaire. 

The  respondents  represent  a  wide  spectrum  of  institutions  that 
provide  different  types  of  financial  management  education  and 
training.  In  terms  of  military  sponsorship,  five  respondents  were 
components  within  or  sponsored  by  the  Department  of  the  Army,  three 
within  the  Department  of  the  Air  Force,  and  eleven  by  the 
Department  of  the  Navy  (including  one  response  from  the  Marine 
Corps)  . 
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OF  PINAKCIAL  MANAGEMENT  EDUCATION  AND  TRAINING  PROVIDERS 


GENERAL  CHARACTERISTICS 

MO.  OF  RESPONSES 

MILITARY  SPONSORSHIP; 

ARMY 

5 

AIR  FORCE 

3 

NAVY 

11 

LEVEL  OF  PROGRAM; 

ENTRY  LEVEL 

1 

PROFESSIONAL 

9 

FULL  SPECTRUM 

9 

AVERAGE  ENROLLMENT; 

LESS  THAN  FIFTY 

2 

FIFTY  TO  ONE  HUNDRED 

3 

ONE  HLTIDRED  ONE  TO  TWO  HUNDRED 

2 

MORE  THAN  TWO  HUNDRED 

10 

NUMBER  OF  COURSES  OFFERED; 

ONE 

5 

TWO  TO  FIVE 

4 

SIX  TO  TEN 

4 

MORE  THAN  TEN 

6 

TARGET  AUDIENCE; 

MILITARY 

3 

CIVILIAN 

1 

BOTH 

15 

FACULTY  COMPOSITION; 

MILITARY 

4 

CIVILIAN 

4 

BOTH 

11 

In  addition  to  representing  all  of  the  military  services, 
the  respondents  span  the  educational  spectrum  from  those  providing 
basic  military  entry  level  training  to  those  awarding  Master 
degrees.  Nine  of  the  nineteen  respondents  were  classified  as 
providing  professional  level  education  (e.  g. ,  serving  higher 
level  civilian  and  military  personnel  or  providing  an  advanced 
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educational  degree.  Of  these  nine,  five  offer  graduate  level 
programs  leading  to  a  master's  degree. 

In  terms  of  the  size  of  the  institutions  responding  to  the 
survey,  average  annual  enrollment  in  programs  or  courses  ranged 
from  a  low  of  36  to  a  high  of  1,670.  The  total  annual  student 
population  reported  by  the  respondents  was  7,324.  Another  measure 
of  size  is  the  number  of  courses  offered.  The  range  of  courses 
offered  by  institution  was  from  one  to  36,  with  an  average  of 
eight.  The  target  audience  as  reported  by  the  survey  respondents, 
not  surprisingly,  consists  of  both  military  and  civilian  employees 
of  DoD.  Only  one  agency  reported  its  audience  as  civilian  only, 
and  three  responded  military  only.  Fifteen  agencies  serve  both. 

In  terms  of  faculty  composition,  e.  g. ,  military, 
civilian  or  a  combination  of  both,  four  agencies  used  civilian 
instructors,  four  military  and  eleven  had  a  mixture  of  military  and 
civilian. 

B.  Quality  Assurance  Methods  Employed  By  Providers  of 
Financial  Management  Education  and  Training 

The  survey  questionnaire  was  designed  to  elicit  information  on 
the  quality  assessment  methods  used  by  financial  management 
education  and  training  providers.  Since  the  survey  involved  self- 
assessment  of  these  methods,  the  responses  provided  represent  only 
the  views  or  opinions  of  the  providers  on  quality  assurance. 

No  attempt  was  made  to  force  the  responses  into  a  preconceived 
model  of  quality  assurance.  Since  the  survey  is  descriptive,  the 
responses  by  providers  of  financial  management  education  and 
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training  represent  their  interpretation  of  what  quality  assurance 
consists  of  and  the  importance  they  attach  to  the  methods  used  to 
assure  the  quality  of  their  courses  and  programs. 

C.  SURVEY  RESPONSES  BY  QUESTION 

This  section  presents  the  responses  to  the  survey  questions 
organized  by  question.  Those  questions  that  required  a  "YES"  or 
"NO"  response  are  indicated  by  an  asterisk  preceding  the  question 
number.  Each  question  is  followed  by  a  summary  of  the  responses. 
Comments  provided  by  survey  respondents  are  also  included. 

1.  What  Is  the  target  audience  for  vour  proareun? 


The  response  option  for  this  question  was  civilian,  military 
or  both. 


RESPONSE 

NUMBER 

PERCENTAGE 

CIVILIAN 

1 

5.3  % 

MILITARY 

3 

H 

• 

09 

BOTH 

15 

78.9  % 

TOTAL  RESPONSES 

19 

100  % 

While  the  vast  majority  of  respondents  replied  "BOTH",  the 
majority  of  students  are  military.  For  example,  several  agencies 
replying  "BOTH"  reported  military  percentages  in  the  90  percentile 
range.  The  sole  agency  reporting  only  "CIVILIAN"  was  American 
University.  American  University  offers  a  graduate  level  program 
for  civilian  employees  of  the  Department  of  the  Navy. 
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2 


What  ar«  the  aoproaciaaf  PTcentaaes  of  class  attendees? 


The  responses  to  this  question  varied,  but  as  noted  above  the 
majority  of  students  are  military.  Based  on  the  19  surveys 
received,  approximately  80  %  of  all  reported  students  are  military. 

3.  What  are  the  ranges  of  grades  or  ranlcs  of  vour  attendees? 


GRAOE/RANK 

NUMBER 

PERCENTAGE 

ENTRY  LEVEL 

1 

5.2 

"PROFESSIONAL" 

9 

47.4 

FULL  SPECTRUM 

9 

47.4 

TOTAL 

19 

100.0 

Of  the  19  responses  received,  nine  provide  financial 
management  education  for  "professionals".  Professional  in  this 
analysis  was  defined  as  GS/GM  7's  and  above  or  military  officers. 
Only  one  respondent  provided  services  for  junior  enlisted  or 
civilian  personnel.  The  balance  of  providers  offer  financial 
management  education  across  the  full  spectrum  of  civilian  and 
military  pay  grades. 
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4 


How  Many  aeoarate  coxirses  in  financial  management  are 
offered  within  vour  program? 


COURSES 

NUMBER 

PERCENTAGE 

1 

5 

26.3 

2-5 

4 

21.1 

5-10 

4 

21.1 

o 

H 

A 

6 

31.5 

The  range  of  courses  offered  as  reported  by  survey  respondents 
was  from  one  to  36.  The  responses  indicate  that  providers  of 
financial  management  education  and  training  have  some  difficulty 
defining  what  constitutes  courses.  This  question  was  designed  to 
identify  the  number  of  individual  courses  versus  the  number  of 
programs  offered.  It  is  clear  from  the  survey  data  that  some 
consistency  is  present  in  course  definition. 

For  example,  based  on  the  survey  responses,  the  total  number 
of  courses  reported  was  163.  Based  bn  information  previously 
collected  this  number  was  expected  to  be  181.  Further  analysis  is 
required  to  understand  the  cause  of  this  discrepancy.  The  most 
likely  cause  is  the  difficulty  agencies  have  in  identifying  or 
categorizing  specific  courses  as  "financial  management". 
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5. 


What  is  the  length  of  vour  program  in  davs,  veeks  or 
months? 


1  LENGTH 

PERCENTAGE 

1  5  DAYS  OR  LESS 

4 

23.5 

1  ONE  MONTH 

2 

00 

• 

H 

1  1  MONTH  -  1  YEAR 

4 

23.5 

1  ONE  YEAR  OR  MORE 

7 

41.2 

1  TOTAL  RESPONSES 

17 

100.0 

This  question  again  points  out  the  difficulty  in  the 
distinction  between  "courses”  and  "programs".  While  in  some  cases 
they  may  be  the  same,  e.  g. ,  the  course  and  program  constitute  the 
only  offering  at  an  agency  or  department,  the  majority  of  survey 
respondents  replied  on  the  length  of  courses  versus  programs. 


6.  How  many  times  per  year  is  it  offered? 


1  FREQUENCY 

NUMBER 

PERCENTAGE 

1  ONCE 

3 

20.0 

1  TWO  TO  FIVE 

6 

40.0 

1  SIX  TO  TEN 

2 

13.3 

1  MORE  THAN  TEN 

4 

26.7 

1  TOTAL  RESPONSES 

15 

100.0 

12 
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What  is  thm  annual  wrollaent  in  yovlt  program? 


PERCENTAGE  I 

1  LESS  THAN  50 

2 

11.8 

1  50  TO  100 

3 

17.6 

1  101  TO  200 

2 

11.8 

1  MORE  THAN  200 

10 

58.8 

1  TOTAL  RESPONSES 

17 

100.0 

The  average  (mean  )  enrollment  based  on  the  seventeen 
responses  was  431  students.  The  range  of  students  was  from  36  to 
1,670.  The  annual  total  student  population  reported  by  the 
responding  agencies  totalled  7,324. 


8.  What  is  the  average  tmeani  class  size? 


1  SIZE 

NUMBER 

PERCENTAGE 

1  TEN  OR  LESS 

1 

5.3 

1  TEN  TO  TWENTY 

7 

36.8 

TWENTY  TO  THIRTY 

9 

47.4 

MORE  THAN  THIRTY 

2 

10.5 

TOTAL  RESPONSES 

19 

100.0 

The  average  (mean)  class  size  as  reported  by  respondents  was 
23.4.  The  range  was  from  9  to  60  students. 
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9 


If  vour  program  provid9s  course  credit  for  college,  hov 
many  credits  are  offered  in  the  total  program? 


COLLEGE  CREDIT  GRANTED 

NUMBER 

PERCENT 

MASTER  DEGREE 

5 

26.3 

CREDITS  ONLY 

3 

15.8 

NO  COLLEGE  CREDIT 

11 

57.9 

TOTAL  RESPONSES 

19 

100.0 

Eight  responses  were  received  on  this  question.  Of  the  eight 
agencies  responding,  five  were  degree  granting  at  the  Master  level 
with  graduate  credits  ranging  from  42  to  85.  Three  other  agencies 
offered  college  credits  of  6  hours,  16  hours,  and  from  3  to  16 
hours  respectively. 

*10.  Do  students  completing  the  program  receive  a  degree  or 
certificate? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

17 

89.5 

NO 

2 

10.5 

TOTAL  RESPONSES 

19 

100.0 

If  ves^  what  is  the  title  of  this  award? 

Of  the  19  agencies  responding  to  this  question,  ten  issue 
certificates  of  completion,  two  issue  certificates  of  training,  one 
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issue  a  diploma  and  five  issue  masters  degrees.  Two  agencies  do 
not  provide  a  degree  or  certificate  to  graduates.  The  total 
exceeds  19  due  to  one  multiple  response. 


ll<  How  do  YOU  determine  need  for  new  courses  or  instruction? 


The  following  comments  were  provided: 


•  Coordination  with  the  Defense  Finance  and  Accounting  Service; 
Coordination  with  the  Comptroller  of  the  Army  or  Assistant 
Secretary  of  the  Army  for  Financial  Management;  Coordination 
with  the  Army's  Training  and  Doctrine  Command;  and  comments 
from  the  field. 

•  Changes  in  the  operating  environment  in  DoD. 

•  Feedback  from  participants;  initiatives  by  faculty  and 
directors  of  programs;  guidance  from  the  Assistant  Secretary 
of  the  Army  for  Financial  Management. 

•  Assigned  course  proponents  assess  preliminary  needs  prior  to 
an  annual  market  survey  which  determines  definitive  needs. 

•  New  course  start-up  requirements  are  detailed  in  Army 
regulation  350-3  and  are  in  the  updated  Training  and  Doctrine 
Command's  Training  Requirements  Analysis  System. 

•  Inputs  by  users  and  field  activities. 

•  New  course  requirements  are  identified  by  the  customers, 
usually  through  the  use  of  Air  Force  form  19  -  "Request  to 
Establish  New  Continuing  Education  Short  Course".  Biannual 
curriculum  reviews  with  DoD  customers,  and  proceeding  of 
functional  boards  can  also  identify  needs. 

•  Observation  of  changes  in  governmental  and  academic  practices 
and  problems.  Consultation  with  the  program  sponsor. 

•  Communication  with  the  Navy's  contract  officer  and  adjunct 
professors  who  are  professionals  in  the  field. 

•  New  course  and  instructional  needs  are  determined  by  on-going 
feedback  from  students,  aliimni,  faculty  and  our  educational 
service  officers. 
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•  Needs  survey  of  major  claimants,  the  career  planning  board  and 
participation  in  financial  management  professional 
associations. 

•  Fleet  inputs . 

•  Student  feedback  plus  input  from  the  sponsor  of  the  course. 

•  Input  from  the  claimant;  review  of  changes  to  official 
directives;  input  from  field  activities;  input  from  class 
graduates;  input  from  DoD;  review  of  civilian  sector 
approaches . 

•  Fleet  needs;  taskings  from  higher  authority;  critiques  from 
other  courses. 

•  Program  changes;  student  and  claimant  feedback. 

•  Validated  demands  from  field  organizations  for  skills 
training. 

•  The  normal  cycle  for  determination  of  a  new  course  of 
instruction  begins  with  a  change  to  operation  in  the 
community.  The  school  will  get  input  from  headquarters  that 
a  task  or  course  needs  instruction.  Initially,  we  get  input 
from  post-graduation  surveys  which  indicate  that  new 
instruction  is  needed.  Frequently,  we  distribute  field 
surveys  for  more  inputs. 


12.  How  do  YOU  determine  the  need  to  update  or  modify  the 

instruction  you  currently  offer? 


The  following  comments  were  provided: 


•  Changes  in  regulations  or  new  directives;  changes  in  computer 
programs  related  to  financial  management;  lessons  learned,  e. 
g. ,  Desert  Storm;  results  of  enlisted  and  officer  development 
tests . 

•  Changes  in  regulations,  directives  or  operating  environment. 

•  Both  courses  have  a  two-dimensional  evaluation  program, 
subjective  student  end-of-course  and  instructor  or  facilitator 
evaluations.  Courses  are  periodically  reviewed  by  the  course 
proponent,  course  manager  and  training  quality  assurance 
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representative.  Proponents  provide  input  on  regulatory  and 
policy  changes. 

•  We  conduct  post-graduation  surveys  with  graduates  and  their 
supervisors  biannually  to  assess  customer  satisfaction. 

•  Surveys,  proponent  inputs,  MACOM  inputs. 

•  Input  from  field  or  functional  managers;  system  changes;  data 
provided  from  occupational  surveys. 

•  There  are  many  possibilities.  The  faculty  is  responsible  for 
maintaining  currency  in  their  field,  as  the  field  changes,  so 
does  the  course.  Other  sources  are  end-of-course  critiques, 
post-course  critiques,  curriculum  reviews  and  communication 
with  practitioners  and  sponsors. 

•  Current  issues  in  financial  management  plus  guidance  from 
senior  military  and  DoD  comptrollers. 

•  Mid-course  and  end-of-course  student  evaluations. 

•  In  order  to  determine  the  need  to  update  or  modify  the 
instruction  currently  offered,  we  test  alternative  delivery 
methods  of  instruction. 

•  Interaction  with  students,  instructional  staff  and  policy 
makers  from  respective  areas  of  subject  matter. 

•  Annual  course  reviews  and  periodic  curriculum  reviews 
conducted  by  fleet  or  type  commander  staff. 

•  Student  feedback,  major  claimant  input  along  with  changing 
issues  in  financial  management. 

•  End-of-course  critiques  and  surveys  of  past  graduates. 

•  Course  review;  changes  in  reference  material;  changes  in  fleet 
needs . 

•  Continuous  review  of  instructors  and  student  critiques. 

•  The  main  determining  factor  for  modifying  our  instruction  is 
a  change  in  the  regulations  governing  financial  management. 
Post-graduation  feedback  questionnaires  and  several  forms  of 
internal  feedback  also  help  to  determine  when  updates  need  to 
be  made. 
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*13 


.  DO  YOU  hav  onm  or  morm  Bponsors  or  cllants  that  directly 


influence  vour  program  of  instruction? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

18 

94.7 

NO 

1 

5.3 

TOTAL  RESPONSES 

19 

100.0 

If  ves.  wbo  is/are  the  aponsorCs)? 


The  following  sponsors  were  specifically  mentioned  by  survey 
respondents : 

•  Secretary  of  the  Air  Force  for  Financial  Management 

•  Defense  Finance  and  Accounting  Service 

•  Chief  of  Naval  Operations 
«  Navy  Comptroller 

•  Chief  of  Naval  Education  and  Training 

•  Navy  Supply  Systems  Command 

•  Fiscal  Division,  Headquarters,  Marine  Corps 

•  CINCLANTFLT,  COMNAVAIRLANT ,  CNO 

•  Navy  Bureau  of  Personnel 

•  DoD  Comptroller 

•  Navy  Bureau  of  Medicine 

•  NAVFAC 

•  Defense  Acquisition  University 

•  NCD,  NCB,  CFMCP 

•  Navy  Exchange  Command 
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Navy  Food  Service  Systems  Command 


*14.  Is  course  development  guided  or  directed  bv  higher  authority? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

13 

68.4 

NO 

6 

31.6 

1  TOTAL  RESPONSES 

19 

100.0 

If  so.  who? 

•  Soldier  Support  Center,  CASCOM,  TRADOC,  ASA  (FM) ,  DFAS 

•  Assistant  Secretary  for  the  Army  for  Financial  Management 

•  Corp  of  Engineers  Training  Issues  Committee 

•  Headquarters,  Air  Training  Command 

•  OP-82/Director  of  Office  of  Budgets  and  Reports,  NCB 

•  NCD,  NCB,  NCF 

•  Chief  of  Naval  Technical  Training 

•  CNET  and  NAVFAC 

•  BUMED 

•  COMNAVAIRLANT 

•  Standards  Branch,  Narine  Corps 


15.  What  types  of  guidance  do  you  receive? 


The  following  comments  were  provided: 
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•  Regulations  plus  directed  common  core  training. 

•  Learning  objectives. 

•  Feedback  from  collective  leadership  and  periodic  assessment. 

•  Proponents  provide  guidance  on  and  approve  the  technical 
content . 

•  Regulations  and  policy  guidance. 

•  Subjects  of  concern  are  the  appropriate  content  for  courses 
and  the  levels  of  learning  required  for  each. 

•  General  guidelines  as  to  curriculum  content. 

•  Subject  matter  expertise. 

•  Selected  topic  material  or  requirements  for  graduate  skill 
levels. 

•  Course  review,  on-site  evaluations;  constant  feedback  is 
received  from  sponsors. 

•  Curriculum  reviews,  directed  input,  issuance  of  new 
directives. 

•  Formal  instructions. 

•  Internal  curriculum  review. 

•  The  Marine  Corps  uses  the  Instructional  Systems  Development 
process  known  as  the  Systems  Approach  to  Training  (SAT)  as  a 
guideline  for  all  course  development. 

16.  How  often  do  you  receive  such  guidance? 

The  typical  response  to  this  question  was  that  guidance  was 
received  whenever  necessary.  Fourteen  of  the  respondents  answered 
in  this  fashion.  Other  responses  were  annually  (1)  ,  quarterly  (l)  , 
and  two  to  three  times  a  year  (1). 


20 


*17.  DO  YOU  have  ataff  dedicated  to  program  development? _ 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

12 

66.7 

NO 

6 

33.3 

TOTAL  RESPONSES 

18 

100.0 

If  VOS,  how  nanv? 

The  answers  to  this  question  varied  from  one  to  fifty-eight. 
The  average  response  was  between  1  and  3. 


18.  What  are  the  number  of  and  percentages  of  civilian  and/or 

military  Instructors  in  /our  program? 

Of  the  sixteen  agencies  providing  detailed  information  on  this 
question,  the  following  cumulative  data  were  obtained: 

CIVILIAN  Number  112  Percentage  47.5 

MILITARY:  Number  122  Percentage  52.5 

The  range  of  civilian  employees  is  from  0  to  66;  for  military 
employees  the  range  is  0  to  33.  The  average  (mean)  number  of 
civilian  employees  is  6.3.  The  average  (mean)  number  of  military 
employees  was  13.1. 
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19 


What  policies  and  criteria  are  used  to  assign  or  hilt 

■ilitftiTY  ffl<zttXtY/iBg.trvgt9rg  f<?r  Y9vr  prpqrm? 


20. 


The  following  responses  were  provided: 


The  officer  or  enlisted  finance  assignment  branches  screen  and 
submit  records  of  the  best  availedjle  personnel  to  the 
commandant  who  makes  the  final  decision. 

Subject  matter  expertise,  previous  schooling,  previous 
assignments,  and  enlisted/officer  fitness  reports  are  all 
included  as  part  of  the  review  prior  to  actual  assignment. 

Grade,  educational,  experience  and  qualifications. 

Military  candidates  are  screened  by  a  five  member  faculty 
committee. 

A  Master's  degree  is  required.  Broad  knowledge  of  the 
comptroller  field  with  broad  background  in  at  least  one 
functional  area.  Strong  performance  record. 

Must  have  graduate  degree  and  are  financial  management 
specialists. 

Must  have  subspecialty  code;  and  0-4  and  above. 

Manpower  efficiency  reviews  conducted  by  Chief  of  Naval 
Education  and  Training. 

Recent  academic  background  and  field  experience. 

NEC  9502. 

New  Instructors  are  assigned  on  the  basis  of  their  technical 
knowledge  of  the  subject  matter  they  will  teach  and  their 
communication  skills. 


What  policies  and  criteria  are  used  to  appoint  or  hire 
civilian  facultv/lnstruotors  for  vour  program? 


The  following  comments  were  provided: 
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•  All  civil  service  positions  within  the  training  instructor 
field  have  prescribed  niniiaun  qualifications. 

•  College  degree,  subject  matter  expertise,  ability  to  teach. 

•  Coordination/consultation  with  department  chairs  and  academic 
directors. 

•  Instructors  are  recommended  by  their  supervisors. 

•  Review  of  job  series  and  job  experience.  Depending  on  grade, 
career  field  experience  is  generally  required.  Final 
selection  is  based  on  educational  requirements,  subject  matter 
knowledge,  experience,  interview,  and  potential  to  perform  on 
the  "platform”. 

•  Grade,  educational  experience,  qualifications  and  0PM 
guidance . 

•  Civilian  candidates  are  screened  by  a  five  member  faculty 
committee . 

•  Must  have  doctoral  degree.  For  senior  faculty,  must  also  have 
scholarly  research  record. 

•  Resumes,  references,  interviews. 

•  Level  of  degree,  professional  field  experience  in  finance. 


*21*  _ professional  qualifications  standards  required  for 

fagmtY/ingtr\tgt9rg _ (e.a.>  degrees, _ prior _ experience, 

ooppletion  of  an  instructor  training  program)? 


RESPONSE 

hbsssshhi 

YES 

17 

94.4 

NO 

1 

5.6 

TOTAL  RESPONSES 

18 

100.0 

If  ves.  Please  describe. 


•  College  degree. 
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•  Same  standards  as  for  the  University. 

•  Most  instructors  have  a  degree  in  their  subject  area  (minimum 
of  bachelor's)  and  not  less  than  five  years  experience  in 
their  subject  area.  Further,  instructors  must  complete  the 
Instructional  Methods  Course. 

•  All  military  officers  have  degrees;  civilians  usually  have 
degrees.  The  Faculty  Development  Course  must  be  completed 
within  90  days  of  assignment. 

•  Associates  degree  or  higher;  completion  of  Technical  Training 
Instructor  Course. 

•  All  faculty  must  have  an  appropriate  Master's  degree  and 
experience  in  the  field  taught  or  a  closely  related  field. 
Faculty  hired  without  significant  prior  teaching  experience 
must  complete  Academic  Instructor  School. 

•  Master's  degree  plus  completion  of  Air  Force  Academic 
Instructor  School. 

•  We  prefer  faculty  with  Phd's  and  teaching  experience. 

•  Each  faculty  member  in  the  teaching  discipline  at  the  master's 
degree  level  will  hold  the  terminal  degree  in  the  teaching 
discipline  or  a  related  discipline.  Outstanding  professional 
experience  plus  a  Master's  degree  is  preferred. 

•  Specific  Navy  enlisted  classification  codes  apply  to  some 
billets  which  require  fleet  experienced  personnel  to  be 
detailed  to  the  billet. 

•  Minimum  of  Bachelor,  desired  Master's  and  five  years  of 
experience. 

•  Instructor  Training  School;  subspecialty  designation. 

•  Graduate  of  Instructor  training  school. 

•  Prior  experience  in  the  field. 

•  All  instructors  complete  a  two  week  instructor  Training 
Course . 


24 


*22.  Do  YOU  use  classroom  observation  to  evaluate  instruction? 


1  RESPONSE 

NUMBER 

PERCENTAGE 

1 

17 

89.5 

1 

2 

10.5 

1  TOTAL  RESPONSES 

19 

100.0 

If  ves,  who  performs  this  observation? 

The  following  responses  were  provided: 

•  Department  directors,  division  chiefs,  course  directors, 
commandant . 

•  Staff  and  faculty  development  representatives. 

•  Peers . 

•  Program  directors . 

•  Classroom  courses  are  evaluated  by  the  course  manager,  quality 
assurance  representative,  proponent  or  lead  instructor. 

•  School  deans,  department  chairman,  course  directors. 

•  Instructor  supervisors  and/or  flight  commander. 

•  Department  head . 

•  Commandant . 

•  Regional  academic  director. 

•  Educational  specialists. 

•  Staff  instructors. 

•  Academic  Standards  department. 

•  Academic  Program  division. 

•  Qualified  instructors. 

•  Training  director  and  department  head. 
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*  Master  course  manager. 

•  Curriculum  Standards  Branch  Officer,  Instructional  Operations 
Officer,  Academic  Officer,  Academic  Chief,  Instructional 
Operations  Chief. 


How  often  is  It  performed? 


The  following  responses  were  provided: 

•  Weekly  -  5  responses 

•  Quarterly  -  2  responses 

•  Twice  yearly  -  4  responses 

•  Annually  -  6  responses 

•  Unscheduled  or  as  appropriate  -  2  responses 
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*23.  Are  othT  m»thod8  used  to  evaluate  classroom  Instruction? 


1  RESPONSE 

NUMBER 

PERCENTAGE 

YES 

15 

83.3 

NO 

3 

16.7 

TOTAL  RESPONSES 

18 

100.0 

If  vea.  Please  describe. 


Of  the  fifteen  agencies  which  responded  to  this  question,  all 
cited  student  critiques  as  a  method  used  to  evaluate  classroom 
instruction.  Detailed  responses  are  listed  below: 


•  Student  evaluation  of  training. 

•  End-of-class  evaluation  by  students. 

•  Surveys  from  participants;  group  assessment  sessions. 

•  End-of-course  questionnaires  by  students; audits  of  classes  by 
proponent  representatives;  post-graduation  surveys  sent  to 
graduates  and  their  supervisors  6  months  after  course 
completion;  academic  excellence  analysis. 

•  ATC  form  736;  student  critiques;  Training  evaluation  Reports; 
Training  Quality  Reports  from  graduates  and  supervisors  of 
graduates;  field  visits;  STAN  evaluations  and  Inspector 
General  Inspections. 

•  We  use  student/supervisor  post  course  critiques.  The  student 
class  leader  debriefs  the  department  head  at  the  conclusion  of 
each  course. 

•  Student  opinion  questionnaires. 

•  Mid-course  and  end-of-course  student  evaluations. 

•  Student  feedback  instruments;  discussions  with 
instructors/administrators . 

•  Student  feedback  from  every  class.  Representatives  from 
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sponsor  also  monitor  classes. 

•  Student  critiques;  peer  review. 

•  Course  reviews;  student  critiques;  external  evaluations 
(surveys) . 

•  Quarterly  improvement  form  completed  by  all  students. 

•  Course  evaluation  upon  student  course  completion  and  a 
supervisor's  evaluation  of  student  performance  in  training 
skills  3  months  after  course  completion. 

•  Every  instructor  completes  an  after  instruction  report  to  note 
any  problems  with  lesson  materials.  Each  student  fills  out  an 
end-of-course  critique  prior  to  graduation  which  enables  him 
to  evaluate  the  overall  course  material. 


*24.  Are  other  methods  used  to  evaluate  instructors? 


1  RESPONSE 

NUMBER 

PERCENTAGE  | 

YES 

12 

63.2 

NO 

7 

36.8 

TOTAL  RESPONSES 

18 

100.0 

It  Ytgi-BlWt 


The  following  responses  were  received: 


•  Selection  of  instructor  of  the  month,  •  quarter  and  year. 

•  Survey  of  a  sample  of  students  and  their  supervisors  conducted 
six  months  after  course  completion  to  determine  adequacy  and 
application  made  of  training. 

•  End-of-course  questionnaires;  performance  appraisals;  Informal 
visits  to  classroom  by  course  directors. 

•  Student  critiques;  STAN  evaluations;  Inspector  General 
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Inspections . 

•  The  student  course  critique  covers  the  instructor's 
performance . 

•  Student  opinion  questionnaires. 

•  Faculty  may  choose  a  15  minute  tape  presentation,  arrange  a 
supervisory  visit  and  evaluation  or  submit  a  representative 
portfolio  for  evaluation  purposes. 

•  Student  feedback  instruments.  Follow-up  discussions  with 
students/ administrators . 

•  student  critiques;  peer  review. 

•  Course  review;  student  critiques;  external  evaluations. 

•  Review  of  instructor  prepared  course  material. 

•  100  percent  of  the  students  of  each  class  complete  an 
Instructor  Rating  form.  This  form  allows  them  to  evaluate  the 
lesson  material  as  well  as  the  effectiveness  of  the 
instructor. 


•25.  Do  YOU  employ  productivity  measures  to  evaluate 
instructors? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

3 

15.8 

NO 

16 

84.2  1 

TOTAL  RESPONSES 

19 

100.0  1 

If  f9s  Atifirltetj 


Only  three  of  the  nineteen  respondents  indicated  that 
productivity  measures  were  employed.  Of  these  three  only  one 
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respondent  provided  measures  employed.  These  were: 
instructor  contact  hours;  student  training  years  per  instructor; 
instructor  contact  hours  per  student  training  years  produced;  and 
consulting  hours. 


*2«.  Is  vour  institution  accredited? 


RESPONSE 

IlHBSSiiHH 

PERCENTAGE  | 

YES 

9 

47.4 

NO 

10 

52.6 

TOTAL  RESPONSES 

19 

100.0 

If  Yss^  bv  whom? 


•  North  Central  Association  of  Colleges  and  Universities, 
Commission  on  Institutions  of  Higher  Education. 

•  AACSB  for  Schools  of  Management. 

•  Southern  Association  of  Colleges  and  Schools  (2  responses) . 

•  North  Central  Association. 

•  American  Council  on  Education  (2  responses) . 

•  Western  Association  of  Schools  and  Colleges. 

•  Middle  States  Association. 


How  Often? 

Frequency  varied  from  annually  to  every  ten  years. 
Typically,  the  review  cycle  is  five  years. 
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*27.  IS  YQur  Institution  reviewed  bv  other  agencies? 


RESPONSE 

PERCENTAGE 

YES 

13 

68.4 

NO 

6 

31.6 

TOTAL  RESPONSES 

19 

100.0 

If  vss.  bv  vhom  and  how  often? 
Responses  received  included: 


•  Army  Training  and  Doctrine  Command. 

•  American  Council  on  Education  (2  responses) . 

•  Defense  Finance  and  Accounting  Agency;  Secretary  of  the  Air 
Force  for  Financial  Management;  Air  Force  Audit  Agency; 
Community  College  of  the  Air  Force. 

•  Air  University  (2  responses). 

•  National  Association  of  Schools  of  Public  Affairs  and 
Administration . 

•  MIVER  and  QES. 

•  Commander,  Training  Command,  U.  S.  Atlantic  Fleet. 

•  CNET;  CNTECTRA;  NAVFAC. 

•  Navy  and  Bureau  of  Medicine  Inspector’s  General. 

•  Type  commander. 

•  Command  inspections. 
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What  is  the  nature  and  extent  of  this  review? 


The  following  responses  were  provided: 


•  Standard  ACE  review  of  course  material  and  lesson  plans. 

•  Courses  and  library  support  are  evaluated  by  ACE  teams  of 
subject  matter  experts.  Evaluators  consider  factors  such  as 
course  objectives,  subject  matter,  level  of  difficulty, 
duration,  methods  of  student  evaluation,  applicability  to 
post-secondary  education  programs,  and  background  and 
selection  of  students  and  instructors.  ACE  then  makes 
recommendations  to  colleges/universities  for  granting  academic 
credit. 

•  To  determine  accuracy  and  currency  of  training. 

•  Primarily  focuses  on  management. 

•  Curriculum  review. 

•  Year-long  self-assessment  followed  by  site  visit. 

•  Formal  Inspector  General  inspections. 

•  Review  of  curriculum  and  adherence  to  directives. 

•  Inspections. 
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MODE  OF  INSTRUCTION 

PERCENT 

LECTURE 

16 

94.1 

DISCUSSION 

10 

58.8 

SIMUIATION 

8 

47.1 

CASE  STUDY 

7 

41.2 

SEMINARS 

2 

11.8 

INDEPENDENT  STUDY 

6 

35.3 

PROJECT  TEAMS 

3 

17.6 

FIELD  TRIPS 

1 

5.9 

LABORATORIES 

2 

l00a2P  » 8 _ 

PERFORHANCED  BASED 

2  11.8  1 

The  typical  responses  to  this  question  included  lectures, 
discussions,  case  studies,  simulation  and  self  or  independent 
study. 

30.  What  peroentaae  of  Instruction  is  delivered  in  each  mode 

Xfiu _ 


Of  the  eleven  institutions  providing  detailed  percentages  on 
modes  of  instruction,  all  cited  lectures  and  discussions  as  the 
predominant  modes  utilized.  On  average,  lectures  represented  40 
percent  of  the  instructional  program;  discussion  represented  20 
percent.  Simulation,  case  study  and  Independent  study  represent 
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approximately  10  percent  each.  The  remaining  10  percent  was 
divided  among  the  modes  reported  in  Question  29  above. 


31.  What  types  of  course  reading  materials  are  used  in  vour 

pr9qrM> — Uua±u _ tfwth99K9i _ puMiehwq _ articles,  pop 

Pffjpial _ documents,  self-generated  materials)? 


Sixteen  agencies  responded  to  this  question.  Thirteen  cited 
DoD  official  documents.  Ten  developed  their  own  (self-generated) 
materials.  And  nine  utilized  textbooks. 

*32.  Is  vour  program  supported  bv  a  library  containing  extensive 
POP  financial  management  related 
afttfgilXg? 


RESPONSE 

PERCENTAGE  | 

YES 

9 

50.0  1 

NO 

9 

50.0 

TOTAL  RESPONSES 

18 

100.0 

Of  the  eighteen  responses  received,  half  noted  that  they  were 
not  adequately  supported  by  a  library  containing  extensive  DoP 
financially  related  materials.  Question  33  and  34  asked 
specifically  about  library  support.  Six  providers  specified 
Improvements  that  would  like  to  see  in  library  support  (See 
Question  33)  and  six  noted  that  improvements  were  planned  in 
library  support  (See  Question  34) . 
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HOW  vould  YOU  like  to  Improve  library  support  for  vour 


Six  responses  were  provided  on  this  question.  They  included: 

•  Establish  a  library  within  our  school. 

•  Be  able  to  Interact  with  some  DoD  instructional  material  data 
base. 

•  Ensure  that  the  library  has  all  required  materials. 

•  Improve  library  support  by  joining  the  Washington  area 
consortium. 

•  Specified  funds  for  financial  texts. 

•  Obtain  sufficient  copies  of  reference  materials. 


•34.  Are  there  improvements  In  vour  library  services  planned? 


1  RE8POH8B 

ihhbsihh 

PERCENTAGE 

1  YES 

6 

35.3 

NO 

11 

64.7 

1  TOTAL  RESPONSES 

17 

100.0 

If  vea.  Please  explain  these  improvements  and 
indicate  when  they  will  occur. 


•  An  effort  is  underway  to  automate  the  Logistics  Library  which 
will  provide  computer  assisted  in-house  services  plus  remote 
dial-in  access.  (Scheduled  for  FY  1993) 

•  Our  library  has  just  assigned  collection  development 
specialists  to  enhance  library  support  for  school  programs. 
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*35.  18  Your  proaran  supported  bv  a  computer  center  or 


laboratory? 


1  RESPONSE 

NUMBER 

PERCENTAGE 

1  YES 

14 

73.7 

NO 

5 

26.3 

TOTAL  RESPONSES 

19 

100.0 

If  no.  vhat  computer  resources  are  eiPBloved_at  vour 


Only  one  response  was  provided  in  response  to  this  portion  of 
the  question;  that  response  indicated  that  only  personal  computers 
were  used. 


*36.  Does  Your  instructional  program  include  oosputer 
exeroises  or  assignments? 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

16 

88.9 

NO 

2 

11.1 

TOTAL  RESPONSES 

18 

100.0 

If  vas.  Please  describe  how  computers  are  utlliged. 
Responses  included: 

•  Standard  Army  Information  Management  System. 

•  Real-world  applications. 
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•  Familiarize  clients  with  computers. 

•  Simulation. 

•  Data  retrieval  exercises;  National  Budget  case. 

•  Self-paced  instruction. 

•  Application  practices. 

*37.  la  vour  program  well  supported  with  respect  to 
administrative  staff,  funding  for  quest 


RESPONSE 

NUMBER 

PERCENTAGE 

YES 

12 

63.2 

NO 

7 

36.8 

TOTAL  RESPONSES 

19 

100.0 

What  improvements  in  support  would  be  desirable? 

Of  the  four  agencies  that  specified  improvements,  all  noted 
additional  funding  for  guest  speakers  would  be  desirable. 


38.  What  methods  are  used  to  determine  the  need  for  vonr 

current  program? 


The  following  responses  were  provided: 


•  The  Army  conducts  annual  surveys  of  all  users  of  Army  training 
programs . 

•  Sponsors. 

■•v 

•  Needs  assessments  are  conducted  by  proponents  prior  to  any 
consideration  of  course  design  or  development.  Proponents 
annually  verify  the  need  for  classroom  courses  and  an  annual 
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market  survey  provided  the  needs  verification  from  the  users. 

.  The  Total  Army  Centralized  Individual  Training  Solicitation 
(TACITS)  is  a  survey  which  is  conducted  annually  for  primary 
information  and  follow-up;  Structure  Manning  Decision  Review 
is  conducted  annually;  the  Training  Resources  Arbitration 
Panel  (TRAP)  process  is  done  monthly;  off-line  memorandxims  can 
be  submitted  once  the  TRAP  closes  for  execution;  curriculum 
council  reviews;  annual  course  proponent  reviews. 

•  Field  evaluations;  Training  Evaluation  reports;  Training 
Review  Analyses;  occupational  surveys. 

•  Curriculum  review  addresses  this  specific  issue.  The  need  is 
also  assessed  by  the  quantity  of  requests  for  any  particular 
course . 

•  The  need  for  resource  management  education  among  military 
services. 

•  Data  on  P-coded  billets  and  other  financial  management 
positions. 

•  Program  reviews  are  conducted  every  five  years. 

•  Alumni  surveys;  student  surveys;  end-of -course  assessments; 
ESO  inputs;  and  faculty  surveys. 

•  NC  and  NCD  guidance;  Career  Program  Planning  Board; 
initiatives  in  the  operational  environment. 

•  Site  surveys  conducted  by  systems  command  sponsors  and  annual 
course  curriculum  reviews. 

•  Student  feedback  and  sponsors*  requirements. 

•  Demand  for  student  placements. 

•  Fleet  need;  tasking  from  higher  authority;  student  critiques. 

•  Review  by  claimant. 

•  Extensive  course  review  by  program  manager  and  field 
personnel . 

•  The  Marine  Corps  uses  the  Individual  Training  Standards  that 
describe  the  specific  tasks  and  knowledge  required.  We  also 
conduct  Course  Content  Review  Boards  to  ensure  the  training 
standards  are  correct. 
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39.  What  _ ft£S _ UfiAd _ tSi _ def  rmlne _ thfl 

ftggMftgYi _ relevance  and  timeliness  of  vour  current 

program? 


The  following  methods  were  cited: 


•  Annual  review  of  training  strategy  by  subject  matter  experts; 
annual  review  of  lesson  plans;  needs  analysis. 

•  Proposed  training  program  based  on  TACITS  data  is  reviewed  and 
approved  by  the  Department  of  the  Army. 

•  Accreditation  and  standards  of  graduate  program. 

•  Proponents  are  charged  with  the  responsibility  for  technical 
content.  Proponent/ instructor  meetings  as  well  as  monitoring 
of  courses  are  performed  to  ensure  technical  accuracy, 
timeliness,  etc.  Student  end-of-course  critiques  are 
constantly  monitored  for  student  perceived  requirements  for 
change . 

•  Curriculum  council  reviews;  annual  course  proponent  reviews; 
end-of-course  questionnaires;  Structure  Manning  Decision 
review;  Army  Training  Resource  Requirements  System. 

•  Field  evaluations;  Training  Evaluation  Reports;  Training 
Review  Analysis;  occupational  surveys;  Inspector  General 
inspections;  student  critiques.  Also  a  customer  service 
infoxmation  line  has  been  installed. 

•  Curriculum  reviews;  visits  with  the  sponsor;  student 
critiques;  functional  boards;  and  other  feedback. 

•  Review  by  senior  financial  management  officials;  student 
feedback. 

•  Observation  of  changes  in  governmental  and  academic  practices 
and  problems.  Consultation  with  program  sponsor. 

•  Surveys  and  interviews  with  students,  alumni  and  practitioners 
in  the  field. 

•  All  programs  are  under  continuous  review.  Inputs  from  the 
teaching  site  is  gathered  each  term  to  determine  the  validity 
of  our  current  program. 

•  Subject  matter  experts;  career  board  annual  report,  classroom 
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visits,  surveys  of  the  comnunity. 

•  Annual  course  reviews;  systems  command  assessments  and 
technical  audits. 

•  Sponsor  review;  willingness  of  users  to  reimburse  for  the 
course . 

•  Curriculum  survey  to  past  graduates;  review  by  sponsors. 

•  Course  reviews;  standardization  conferences. 

•  Student/claimant  feedback. 

•  Competent  job  performers  and  subject  matter  experts  review 
course  content  for  validity,  accuracy  and  relevance. 

•  Following  completion  of  each  course,  a  career  content  review 
board  is  conducted  to  ensure  the  material  is  accurate,  timely 
and  relevant. 


durina  and  upon  ppooram  completion 

(e.o..  oassina 

standardized  tests,  written  course  work. 

observation  of 

performance  on  the  iob)? 


Responses  provided  included: 

•  Practical  exercises  (4  responses) 

•  Tests  (14  responses) 

•  Research  papers  (4  responses) 

•  Individual  or  group  projects  (3  responses) 

•  Case  studies  (4  responses) 

•  Oral  presentations  (3  responses) 

•  Passing  national  exams  (2  responses) 

•  Thesis  (1  response) 
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*41.  Do  you  hay  a  fornal  nethod  of  student  course  •valuation? 


1  RB8POM8B 

HHi^ssmn 

PERCENTAQE 

1  YES 

19 

100.0 

1 

0 

0.0 

1  TOTAL  RESPONSES 

19 

100.0 

If  vea.  describe  this  orocoss. 

Of  the  nineteen  responses  received  to  this  question,  all 
indicated  that  a  formal  method  of  student  course  evaluation 
existed.  All  used  a  student  critique  form  for  this  purpose. 

*42.  Are  other  methods  used  to  permit  students  to 
evaluate  vour  instructional  program? 


1 . . 

RESPONSE 

NUMBER 

PBRCENTAOB 

YES 

10 

52.6 

NO 

9 

47.4 

1  TOTAL  RESPONSES 

19 

100.0 

If  vee.  describe  these  methods. 


The  following  comments  were  provided: 

•  The  Commandant  holds  periodic  informal  discussions  with 
students . 

•  Post-graduation  surveys  (3  responses) . 

•  DD  form  1556  is  completed  by  students  and  their  supervisors. 
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•  Student-to-instructor  feedback. 

•  Students  provide  informal  comments  to  their  class  leaders,  who 
can  relay  them  at  end-of-course  debriefs. 

•  Students  are  invited  to  meet  with  the  department  chairman 
periodically  and  can  make  Individual  comments  to  him  as  they 
wish. 

•  Follow-up  with  educational  specialists. 


*43.  Are  vour  students  required  to  pass  anv  standardised 
POP  or  national  eraninations  (e.g.,  CPA,  CMA)? 


RE8P0M8B 

PERCENTAGE 

YES 

1 

5.3 

NO 

18 

94.7 

TOTAL  RESPONSES 

19 

100.0 

The  one  respondent  answering  "yes"  to  this  indicated 
that  students  are  required  to  pass  the  6RE,  GHAT  or  MAT. 

44.  What  percentage  of  students  entering _ ysSL _ program 

suooessfullv  ooaplete  it? 

Of  the  eleven  responses  to  this  question,  10  indicated  that 
completion  rates  averaged  over  95  percent.  One  agency  cited  a 
completion  rate  of  75  percent. 
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*45.  Do  YOU  track  the  perforaance  of  votir  graduates? 


1  RS8PON8B 

PERCENTAGE 

1 

7 

36.8 

1 

12 

63.2 

1  TOTAL  RESPONSES 

19 

100.0 

If  vmm.  how  is  thla  dono? 


Of  the  seven  responses  provided,  six  agencies  cited  the  use  of 
post-graduate  surveys  as  the  principle  method  utilized  to  track 
graduate  performance.  Specific  comments  were: 


•  School  liaison  teams  visit  finance  units  in  the  field  to 
evaluate  school  training  and  training  needs  with  graduates  and 
their  supervisors.  Survey  of  a  sample  of  students  and  their 
supervisors  is  conducted  about  six  months  after  course 
completion  to  determine  adequacy  and  applicability  of 
training. 

•  Post  graduate  surveys  are  sent  to  graduates  and  their 
supervisors  six  months  after  course  completion. 

•  Field  visits  by  training  evaluators. 

•  We  use  one  year  and  five  year  out  surveys  of  graduates. 

•  Follow-up  questionnaire  one  year  after  completion;  survey  of 
supervisors  of  graduates  of  our  program;  track  progression 
through  the  alumni  association. 

•  A  post-course  evaluation  is  sent  to  the  student's  supervisor. 
It  is  intended  to  determine  the  effectiveness  of  training  more 
than  the  performance  of  the  individual. 

•  The  supervisors  of  all  graduates  are  sent  a  feedback 
questionnaire  to  help  us  evaluate  if  the  student  learned  the 
necessary  prerequisite  knowledge. 
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1  RESPONSE 

PERCENTAGE 

1  YES 

12 

63.2 

1  NO 

7 

36.8 

1  TOTAL  RESPONSES 

19 

100.0 

If  v».  how  is  this  don»? 

Of  the  twelve  responses  received,  nine  agencies  cited  the  use 
of  post-graduate  surveys  for  this  purpose.  Post-graduate  surveys 
were  used  for  two  purposes:  (1)  To  track  the  performance  of 
graduates  (see  Question  45  above),  and  (2)  To  assess  the 
effectiveness  or  utility  of  the  program. 

47.  m  Your  opinion,  what  are  the  most  important  methods  you 

♦nplQY  to  ensure  the  quality  of  vour  program? 


This  question  was  designed  to  be  the  cap-stone  question  of  the 
survey.  It  provided  the  opportunity  to  summarize  the  methods  used 
by  survey  respondents  to  ensure  quality.  Responses  included: 


•  Visits  to  finance  units  in  the  field;  selection  of 
instructors/ subject  matter  experts  who  have  field  experience; 
insuring  that  we  are  kept  informed  on  the  numerous  changes  in 
finance,  accounting  and  resource  management  regulations  and 
policies. 
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•  Qualified  faculty;  feedback  from  students  and  their 
supervisors. 

•  Recognition  by  sponsors  and  commands/agencies  which  provide 
participants  for  the  program;  competition  for  individuals  to 
attend;  long  standing  reputation  of  the  program. 

•  Needs  assessment;  task  analysis;  analysis  of  pre/post  test 
results;  quality  assurance;  proponent  and  course  manager 
evaluations;  periodic  proponent/ instructor/SME  workshops  for 
technical  accuracy  and  relevance;  analysis  of  student  end-of- 
course  evaluations. 

•  Close  attention  to  post-graduation  surveys;  Close  attention  to 
comments  on  end-of-course  questionnaires;  continuing  dialogue 
with  students  and  others  in  the  functional  area;  exceptional 
instruction;  extensive  knowledge  of  subject  matter  and 
regulatory  requirements. 

•  Field  visits;  student  evaluations;  rotation  of  military 
Instructors;  field  feedback;  workshops  to  review  training. 

•  We  attempt  to  hire  and  maintain  a  top  quality  faculty.  The 
Faculty  Academics  Standards  Committee  reviews  each  course 
every  three  years. 

•  Constant  review  of  the  curriculum  by  the  faculty  plus  biannual 
review  by  Air  University  and  senior  DoD  comptrollers;  student 
feedback. 

•  Faculty  involvement  with  the  Navy  and  Defense  Department  and 
in  academic  activities  and  organizations. 

•  We  set  high  standards  for  professional  graduate  level  work  and 
expect  the  faculty  to  meet  these  standards. 

•  Student  course  assessments;  faculty  observations. 

•  Review  of  material  for  accuracy  of  content;  student  and 
instructor  evaluations;  post-training  job  performance. 

•  Periodic  fleet/type  commander  staff  reviews  and  systems 
command  technical  assessments. 

•  Constant  feedback  from  the  students  and  inputs  from  the  course 
sponsor. 

•  Collection  of  data  via  student  critiques;  trend  analysis  over 
time. 

•  Fleet  Inputs . 
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•  On-going  student  and  client  feedback. 

•  Feedback  from  students  and  their  supervisors. 

•  The  Instructional  Systems  Development  Process  guides  us  to 
ensure  the  c(uality  of  our  program.  Additionally,  our  school 
enjoys  an  outstanding  relationship  with  field  personnel.  We 
are  able  to  respond  to  changes  to  regulations  and  provide 
better  instruction. 


48.  Is  there  anv  other  information  that  vou  would  offer  on 

quality  assessment  at  vour  institution  that  we  did  not 
request? 

No  responses  were  provided  to  this  question. 

III.  ANALYSIS  OF  SURVEY  RESPONSES 
Analysis  of  responses  begins  with  the  broadest  element  of  the 
survey—  responses  to  question  47,  Question  47  asked —  What  are  the 
most  important  methods  you  employ  to  ensure  the  quality  of  your 
program?  The  question  gave  respondents  the  opportunity  to  summarize 
the  methods  they  used  to  assure  quality.  Responses  to  this 
question  form  the  basis  for  evaluating  the  quality  assurance 
methods  employed  by  financial  management  education  and  training 
providers . 

Quality  Assuranoe  Methods  Employed 

The  following  list  summarizes  methods  cited  by  survey 
respondents  to  ensure  the  quality  of  their  programs: 

•  Student  feedback  (11  responses). 
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•  Sponsor  guidancs/ feedback/ support  (6  responses) . 

•  Hiring/ensuring  quality  faculty  (6  responses). 

•  Feedback/ oontaot  with  clients  (9  responses) . 

•  Tracking  the  post>graduation  performance  of  students  (5 

responses) . 

•  Monitoring  of  faculty  performance  (3  responses). 

•  Keeping  abreast  of  changes  in  financial  management  (4 

responses) . 

•  Tracking  student  performance  in  courses  and/or  programs  (2 
responses) . 

The  methods  noted  by  financial  management  education  and 
training  providers  in  their  responses  to  this  survey  question 
(Question  47)  also  are  supported  by  the  responses  made  to  other 
questions  within  the  survey  questionnaire.  Each  of  the  methods 
cited  above  are  discussed  separately  below  with  references  to  other 
survey  questions  as  appropriate. 

1 .  Student  Feedback 

The  use  of  student  feedback,  specifically  end-of~course 
critique  forms,  is  a  response  that  appears  throughout  the  surveys 
returned  by  financial  management  education  and  training  providers. 
In  addition  to  Question  41  which  asks  —  Do  you  have  a  fotmal 
method  of  student  course  evaluation?  to  which  all  nineteen 
respondents  affirmatively  replied,  the  questions  and  the  number  of 
respondents  that  listed  student  feedback  as  a  prime  determinant  are 
as  follows: 

•  Question  11:  How  do  you  determine  the  need  for  new  courses  or 
instruction?  (6  responses) 
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•  Question  12:  How  do  you  determine  the  need  to  update  or 
modify  the  instruction  you  currently  offer?  (10  responses) 

•  Question  23:  Are  other  methods  used  to  evaluate  classroom 
instruction?  [Note:  Question  23  asks  whether  respondents  use 
classroom  observation  to  evaluate  Instruction]  (15  responses) 

•  Question  24:  Are  other  methods  used  to  evaluate  instructors? 
(7  responses) 

•  Question  38:  What  methods  are  used  to  determine  the  need  for 

your  current  program?  [Follow-on  to  questions  11  and  12]  (3 

responses) 

•  Question  39:  What  methods  are  used  to  determine  the  validity, 
accuracy,  relevance  and  timeliness  of  your  current  program? 
(7  responses) 


2 .  Sponsor  Guidance 

Sponsor  guidance  is  another  common  method  of  quality 
assurance  employed  by  respondents  to  the  survey.  Question  13  (Do 
you  have  one  or  more  sponsors  or  clients  that  directly  influence 
your  program  of  instruction?)  directly  measures  this  practice. 
Eighteen  of  the  nineteen  respondents  acknowledged  sponsor  guidance. 
Question  14  (Is  course  development  guided  or  directed  by  higher 
authority?)  also  addresses  this  practice.  Thirteen  of  nineteen 
respondents  responded  "yes"  to  the  question. 

Additionally,  the  following  survey  questions  applied  to 
the  sponsor  guidance  [  the  number  of  positive  responses  is  shown] 

•  Question  11:  How  do  you  determine  need  for  new  courses  or 
instruction?  (9  responses) 

•  Question  12:  How  do  you  determine  the  need  to  update  or 

modify  the  instruction  you  currently  offer?  (8  responses) 

•  Question  27:  Is  your  Institution  reviewed  by  other  agencies 
[other  than  accreditation]?  (7  responses) 
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•  Question  38:  What  methods  are  used  to  determine  the  need  for 
your  current  program?  (6  responses) 

•  Question  39:  What  methods  are  used  to  determine  the  validity, 
accuracy,  relevance  and  timeliness  of  your  program?  (8 
responses) 


3.  Hiring  Quality  Faculty 

Hiring  of  quality  instructors  or  faculty  was  the  third 
quality  assurance  method  cited  by  survey  respondents.  In  this 
regard,  the  issue  of  faculty  *•  quality”  or  competence  was  addressed 
directly  by  the  survey  in  questions  19  through  25.  Question  21  is 
the  most  specific.  It  asked  -*  Are  professional  qualification 
standards  required  for  faculty/ instructors?  Seventeen  of  18 
respondents  stated  that  they  employed  this  method. 

Also  relevant  for  the  discussion  of  quality  faculty  were 
questions  19  and  20.  The  questions  asked  -*  What  policies  and 
criteria  are  used  to  hire  military  (Question  19)  or  civilian 
(Question  20)  faculty/ instructors  for  your  program.  Survey 
responses  to  both  questions  focused  on  experience,  education  and 
other  factors  that  prepare  individuals  for  the  teaching  profession. 
In  fact  all  respondents  noted  that  they  used  selected  criteria  to 
ensure  the  hiring  of  qualified  individuals. 

Question  22  through  25  further  addressed  the  issue  of 
Instructor  or  instruction  observation  to  promote  quality  faculty. 
The  issue  of  faculty  on-the-job  performance  is  discussed  in 
subsection  (f)  Faculty  Performance. 
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4.  Feedback  from  Clients 

Client  feedback  was  the  fourth  method  cited  by  survey 
respondents  to  ensure  quality.  Question  13  addressed  this  method 
by  asking  —  Do  you  have  one  or  more  sponsors  or  clients  that 
directly  influence  your  program  of  instruction?  As  the  question 
implies,  the  distinction  made  between  clients  and  sponsors  was  not 
always  clear  in  the  responses.  Only  one  respondent  specifically 
mentioned  a  client  that  influenced  the  program.  Nevertheless, 
other  survey  question  responses  noted  the  importance  that  financial 
management  education  and  training  providers  give  to  client 
feedback. 

The  following  questions  applied: 

•  Question  11:  How  do  you  determine  the  need  for  new  courses  or 
instruction?  (9  responses) 

•  Question  12:  How  do  you  determine  the  need  to  update  or 
modify  the  instruction  you  currently  offer?  (3  responses) 

•  Question  38:  What  methods  are  used  to  deteinnine  the  need  for 
your  current  program?  (2  responses) 

•  Question  39:  What  methods  are  used  to  determine  the  validity, 
accuracy,  relevance  and  timeliness  of  your  current  program? 
(3  responses) 

Additionally,  client  feedback  is  obtained  through  the  use 
of  post-graduation  surveys  to  the  extent  that  graduates  and  their 
supervisors  represent  the  clients  of  the  courses  and  programs 
offered.  This  practice  is  more  fully  examined  below. 

5.  Post-graduation  Performance  of  students 

The  post-graduation  performance  of  financial  management 
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education  and  training  students  was  the  fifth  method  of  quality 
assurance  indicated  by  survey  respondents.  This  method  was 
specifically  addressed  in  the  survey  in  Question  45.  The  question 
stated  —  Do  you  track  the  performance  of  your  graduates?  Seven  of 
the  nineteen  respondents  did  utilize  such  tracking.  Of  these 
seven,  six  used  post~graduation  surveys  for  this  purpose. 

6 .  Faculty  Performance 

A  sixth  method  of  quality  assurance  reported  was  faculty 
performance.  This  method  is  similar  to  the  hiring  of  quality 
faculty  previously  discussed  in  subsection  (c)  .  However,  the 
monitoring  of  faculty  performance  is  an  on-going  process  that,  in 
principle,  verifies  the  hiring  policies  and  procedures  of  financial 
management  providers. 

Questions  22  through  25  addressed  the  issue  of  faculty 
performance.  These  questions  with  the  number  of  affirmative 
responses  are  as  follows: 

•  Question  22:  Do  you  use  classroom  observation  to  evaluate 
instruction?  (17  responses) 

•  Question  23:  Are  other  methods  used  to  evaluate  classroom 
instruction?  (15  responses) 

•  Question  24:  Are  other  methods  used  to  evaluate  instructors? 
(12  responses) 

•  Question  25:  Do  you  employ  productivify  measures  to  evaluate 
Instructors?  (3  responses) 

One  of  the  prime  measurements  of  faculty  performance  was 
the  use  of  student  critiques  (end-of -course  evaluations)  and 
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student  feedback.  For  example,  all  fifteen  of  the  responses  to 
question  23  cited  the  use  of  student  critiques  as  the  method  used 
to  evaluate  classroom  instruction.  Similarly,  eight  respondents  to 
question  24  noted  student  critiques  as  the  primary  method  employed 
to  measure  Instructor  performance. 

7.  Keeping  Abreast  of  Changes 

Keeping  abreast  of  changes  in  financial  management  was 
the  sixth  quality  assurance  method  cited  by  survey  respondents. 
Keeping  abreast  of  changes  in  financial  management  was  noted  in  the 
following  survey  responses  by  question: 


•  Question  12:  How  do  you  determine  the  need  to  update  or 

modify  the  instruction  you  currently  offer?  (5  responses) 

•  Question  39:  What  methods  are  used  to  determine  the  validity, 
accuracy,  relevance  and  timeliness  of  your  current  program? 
(2  responses) 


8.  Tracking  student  Performance 

The  final  method  reported  to  ensure  quality  was  the 
tracking  or  monitoring  of  student  performance  while  attending  the 
financial  management  institution.  The  survey  addressed  the  issue  of 
student  performance  or  competence  in  questions  40,  43  and  44. 

Question  40  asked  —  What  methods  are  employed  to 
determine  student  competence  during  and  upon  program  completion  (e. 
g. ,  passing  standardized  tests,  written  course  work,  observation  of 
performance  on  the  job)?  Question  43  asked  about  the  requirement 
for  students  to  pass  national  examinations.  Only  one  respondent 
required  this.  Finally,  Question  45  asked  respondents  what 
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percentage  of  students  entering  the  program  successfully  complete 
it?  The  average  successful  completion  rate  was  in  excess  of  95 
percent. 


Additionally,  while  this  method  is  directed  toward  in- 
house  success,  post-graduation  performance  also  informs  the 
institution  on  the  quality  of  their  program. 

SUMMARY 

The  preceding  sections  analyzed  responses  to  the  survey  of 
financial  management  education  and  training  quality  assurance 
methodology.  Table  IV  summarizes  the  methods  used.  The  percentage 
column  is  calculated  on  the  basis  of  the  number  of  responses 
divided  by  the  total  number  of  responses  received. 


TABLE  IV.  SUMMARY  OF  OUAblTY  ASSORAMCE  METHODS 


1  Method 

Responses 

Percentage  | 

(a) 

Student  Feedback 

11 

• 

VO 

1 

Sponsor  Guidance 

6 

31.6 

1 

Hiring  Quality  Faculty 

6 

31.6 

Feedback  from  Clients 

9 

47.4  1 

1  (e) 

Post-graduation  Performance 

5 

23.3  1 

1 

Faculty  Performance 

3 

15.8  1 

1 

Keeping  Abreast  of  Changes 

4 

21.1  1 

1  (h) 

Tracking  Student  Performance 

2 

in 

• 

o 

H 

ZV.  COMCLUSIOMS 


The  respondents  to  this  study  represent  a  wide  spectrum  of  DoD 
institutions  currently  providing  financial  management  education  and 
training.  Five  institutions  responding  operate  within  or  are 
sponsored  by  the  Department  of  the  Army,  three  by  the  Department  of 
the  Air  Force,  and  eleven  by  the  Department  of  the  Navy. 

The  respondents  span  the  educational  spectrum  from  those 
granting  Master  degrees  to  those  providing  basic  military  entry 
level  training.  In  terms  of  the  size  of  the  institutions 
responding  to  the  survey,  average  enrollment  in  programs  or  courses 
ranges  from  a  low  of  36  to  a  high  of  1,670.  The  range  of  courses 
offered  is  from  one  to  36,  with  an  average  of  eight. 

The  target  audience  as  reported  by  the  survey  respondents  was 
both  military  and  civilian  employees  of  DoD.  Only  one  agency 
reported  its  audience  as  civilian  only,  and  three  responded 
military  only.  The  remainder  serve  both. 

In  terms  of  faculty  composition  (  military,  civilian  or  a 
combination) ,  four  agencies  use  civilian  instructors,  four 
military,  and  eight  have  a  mixture  of  military  and  civilian. 

The  question  addressed  by  this  research  was  the  method  of 
quality  assurance  employed  by  financial  management  providers.  The 
responses  to  the  survey  of  quality  assurance  methods  are  listed 
below. 

(a)  Student  Feedback 

(b)  Sponsor  Guidance 
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(c)  Hiring  Quality  Faculty 

(d)  Feedback  from  Clients 

(e)  Post-graduation  Performance 

(f)  Faculty  Performance 

(g)  Keeping  Abreast  of  Changes  in  DoD  Financial 
Management 

(h)  Tracking  Student  Performance 

Student  feedback  and  feedback  from  clients  were  the  two 

quality  assurance  methods  most  commonly  cited  by  survey 
respondents.  The  survey  results,  however,  clearly  point  out  the 
wide  divergence  of  quality  methods  used  by  financial  management 
instruction  providers. 

In  addition  to  the  methods  cited  by  survey  respondents  to 
ensure  quality,  other  quality  measurement  methods  might  be 
employed.  Accreditation  and  assessment  of  the  quality  of  support 
resources  also  are  quality  assurance  approaches  that  could  be 
employed  by  financial  management  education  and  training  providers. 

(1)  Aooreditation 

An  accreditation  process  provides  an  independent 
evaluation  of  the  performance  of  educational  institutions. 
Accreditation  organizations  typically  review  courses  and  programs 
for  content,  instructor  competence,  and  the  availability  of 
ancillary  services  including  libraries  and  computers.  Question  26 
of  the  survey  asked  -  Is  your  institution  accredited?  Nine  of  the 
nineteen  respondents  replied  affirmatively.  The  accrediting 
organizations  mentioned  included  the  North  Central  Association  of 
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Colleges  and  Universities,  Commission  on  Institutions  of  Higher 
Management,  the  American  Council  on  Education  and  the  Western 
Association  of  Schools  and  Colleges.  Existing  accreditation 
processes  are  not  applicable  to  all  of  the  respondents  to  the 
survey.  However,  accreditation  would  appear  useful  to  ensure  the 
quality  of  all  DoD  courses  and  program  offerings,  if  such  a  process 
accommodated  the  unique  nature  of  DoD  Financial  Management 
education  and  training  institutions. 

(2)  Evaluating  the  Adequacy  of  Institutional  Support 

Another  method  for  ensuring  quality  is  to  assess  the 
adequacy  of  institutional  support  for  instructional  courses  or 
programs . 

The  survey  questionnaire  asked  several  questions  on 
institutional  support: 

•  Question  32:  Is  your  institution  supported  by  a  library 

containing  extensive  DoD  financial  management  related 
materials? 

•  Question  35:  Is  your  program  supported  by  a  computer  center 
or  laboratory? 

•  Question  37:  Is  your  program  well  supported  with  respect  to 
administrative  staff,  funding  for  guest  speakers,  etc.? 

Responses  to  these  question  were  mixed.  Fifty  percent  of  the 
respondents  reported  that  their  program  was  not  supported  by  a 
library  with  extensive  DoD  material.  Seven  of  nineteen  respondents 
answered  that  their  program  was  not  well  supported  with  respect  to 
administrative  staff  or  funding  for  guest  speakers.  Four  responses 
identified  funding  for  guest  speakers  as  the  only  deficiency  in 
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this  area.  Adequate  support  is  necessary  to  assure  quality  of 
instruction. 

While  it  night  seem  obvious  that  developing  and  maintaining 
quality  programs  necessitates  the  involvement  not  only  of  the 
provider  but  also  of  the  sponsor  and  clients,  the  overall  response 
rate  indicates  that  this  is  not  always  the  case.  Furthermore,  the 
differences  in  responses  on  this  issue  suggests  that  more  effort 
needs  to  be  expended  toward  the  development  of  methods  that  ensure 
financial  management  education  and  training  is  relevant  to  both 
sponsors  and  clients. 

V.  RECOMMENDATIONS 

There  is  no  single  "best  method"  to  assure  quality  in  DoD 
financial  management  education  and  training.  A  mix  of  different 
approaches  appropriate  to  the  mission  and  nature  of  instruction 
provided  at  each  institution  is  needed.  Responses  to  the  survey  of 
DoD  FM  instruction  providers  indicate  that  a  number  of  approaches 
to  quality  assurance  are  employed  currently.  However,  there  is 
room  for  improvement.  Based  upon  the  findings  from  this  survey, 
the  following  recommendations  are  made  to  improve  DoD  Financial 
Management  Education  and  Training  at  all  institutions  in  the  DoD 
system. 

1.  All  DoD  FM  education  and  training  institutions  should  employ 
student  evaluation  administered  at  the  mid-point  and  at  end  of  the 
course  of  study  to  provide  feedback  as  to  the  quality  and  adequacy 
of  instruction.  Where  such  evaluational  methods  exist. 
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institutions  should  evaluate  the  instrunents  used  to  collect 
information  from  students  to  determine  whether  the  necessary  and 
correct  information  is  being  solicited  from  students.  Few 
institutions  presently  administer  mid-point  evaluations  presently. 
This  deficiency  should  be  corrected. 

2.  Careful  analysis  of  student  course  evaluations  collected  over 
the  past  three  years,  with  emphasis  on  the  most  recent  data,  should 
be  performed  to  determine  changes  needed  in  instruction. 

3.  All  DoD  FM  education  and  training  institutions  should  develop 
and  Implement  a  system  for  tracking  students  that  have  received 
instruction.  This  system  should  continue  to  track  students  as 
their  employment  location  changes. 

4.  All  OoD  FM  education  and  training  institutions  should  develop 
and  Implement  post-study  student  satisfaction  surveys  to  obtain 
information  from  students  regarding  the  usefulness  of  the  education 
and  training  they  received,  as  well  as  recommendations  for  change. 
Methods  employed  by  selected  institutions  at  present  to  collect 
these  data  should  be  assessed  to  determine  the  best  approach  to 
collection  and  analysis  of  student  responses.  Data  should  be 
collected  and  analyzed  after  the  first  year  following  the  end  of 
instruction  and  then  subsequently  in  a  three  year  cycle. 

5.  All  DoD  FM  education  and  training  institutions  should  develop 
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and  Inplement  post^study  employer  satisfaction  surveys  to  obtain 
information  from  the  employers  of  graduated  students  regarding  the 
usefulness  to  the  employer  of  the  education  and  training,  as  well 
as  recoxnmendations  for  change.  Methods  employed  by  selected 
Institutions  at  present  to  collect  these  data  should  be  assessed  to 
determine  the  best  approach  to  gather  employer  responses.  Data 
should  be  collected  and  analyzed  after  the  first  year  following  the 
end  of  instruction  and  then  subsequently  in  a  three  year  cycle. 

6.  As  noted  above,  no  single  "best  method"  is  feasible  to  assure 
quality  in  DoD  financial  management  education  and  training. 
However,  a  systematic  approach  to  quality  assessment  should  be 
developed  within  DoD  to  assess  quality  on  a  regular  basis  to 
provide  feedback  to  institutions  and  DoD  executives  on  the  quality 
and  adequacy  of  instruction  provided.  Accordingly,  a  single  FM 
education  and  training  accreditation  organization  should  be  created 
to  perform  accreditation  reviews  of  institutions  providing  FM 
instruction.  Accreditation  standards,  guidelines  and  methods  for 
evaluating  compliance  by  institutions  with  these  standards  should 
be  developed.  A  schedule  for  regular  and  periodic  accreditation 
review  including  site  visitation  and  participant  observation  should 
be  established  by  this  organization. 

7.  Quality  assurance  accreditation  reviews  should  be  performed  by 
the  designated  FM  education  and  training  accreditation  organization 
assisted  by  subject  matter  specialists  on  a  three  year  cycle  to 
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assist  institutions  in  maintaining  currency  in  their  curricula, 
instructional  methods  and  technology. 

8.  Accreditation  reviews  of  institutions  should  be  performed  by 
teams  consisting  of  one  member  of  the  accrediting  agency  and  three 
subject  matter  specialists  chosen  from  a  pool  of  experts 
established  for  purposes  of  such  review.  Selection  of  subject 
matter  experts  should  be  based  upon  academic  and  professional 
experience  and  accomplishment  in  the  field  of  instruction  subject 
to  review.  The  pool  of  experts  should  be  drawn  from  within  DoD  and 
from  institutions  external  to  DoD.  All  accreditation  teams  should 
contain  at  least  one  expert  from  a  non-DoD  institution.  To  avoid 
duplicative  review,  where  Institutions  already  have  received 
accreditation  by  recognized  national  or  regional  accreditation 
agencies,  the  DoD  accreditation  review  agency  may  choose  to  accept 
such  reviews  in  lieu  of  performing  its  own  review.  However,  where 
this  is  the  case,  the  results  of  the  accreditation  review  perfoimed 
by  an  external  organization  should  be  examined  by  the  DoD 
accreditation  agent  to  assure  general  conformance  with  DoD  guality 
assurance  standards  and  guidelines. 

9.  The  guality  assurance  accreditation  compliance  process  should 
be  designed  to  include  the  careful  analysis  of  student  course 
evaluations  and  information  from  the  employers  of  graduated 
students  regarding  the  usefulness  to  the  employer  of  the  education 
and  training. 
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10.  The  quality  assurance  accreditation  compliance  process  should 
be  designed  to  assist  Institutions  in  meeting  standard  rather  than 
punishing  them  for  non-compliance.  Resource  augmentation  to 
enhance  library  services,  instructional  technology  and  facilities 
should  be  considered  where  such  aid  appears  to  be  needed.  However, 
institutions  that  fail  to  implement  and  report  on  methods  and 
actions  to  achieve  compliance  with  DoD  standards  and  guidelines 
within  a  reasonable  period  of  time  should  be  considered  for 
budgetary  sanction  and  possible  termination. 

11.  Based  upon  a  review  of  the  missions  of  DoD  financial 
management  education  and  training  organizations,  the  quality 
assurance  accreditation  process  should  be  developed  and 
administered  by  the  Defense  Resource  Management  Institute  advised 
by  a  committee  that  includes  a  single  member  representing  each  DoD 
FH  education  and  training  Institution  plus  selected  subject  matter 
specialists  in  financial  management  education  and  training.  This 
committee  should  meet  semi-annually  or  on  an  "as  needed"  basis  to 
provide  guidance  to  the  accreditation  process. 

12.  Based  on  survey  results  there  is  a  need  to  establish  or 
enhance  financial  management  library  resources  at  selected 
institutions.  This  need  should  be  addressed  as  a  part  of  the 
accreditation  review  process  outlined  above. 

13.  Based  on  survey  results  there  is  a  need  to  establish  or 
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enhance  computer  and  education  technology  resources  at  selected 
institutions.  This  need  also  should  be  addressed  as  a  part  of  the 
accreditation  review  process  outlined  above. 

14.  Based  on  survey  results,  there  is  a  need  to  establish  or 
enhance  resources  to  support  guest  and  visiting  instructors  and 
lecturers  and  speakers  at  selected  institutions.  This  need  should 
be  addressed  as  a  part  of  the  accreditation  review  process  outlined 
above . 

15.  A  pool  of  funds  should  be  created  and  administered  by  the  DoD 
accreditation  agency  to  award  annual  grants  on  a  competitive  basis 
to  institutions  to  establish  or  enhance  library,  computer,  guest 
speaker/lecturer  and  other  services  to  increase  the  quality  of 
financial  management  education  and  training.  Additionally,  a 
competition  should  be  established  for  an  award  for  outstanding 
educational  achievement  by  a  DoO  financial  management  education  and 
training  Institution  to  recognize  and  reward  quality  performance. 
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APPENDIX  A: 


SURVEY  METHODOLOGY 

The  survey  that  was  sent  out  to  all  providers  of  financial 
management  education  and  training  programs  is  included  in  Appendix 
B.  The  list  of  DoD  agencies  and  departments  responding  to  the 
survey  is  provided  in  Appendix  C. 

Prior  tc  developing  and  mailing  the  survey  to  all  providers  of 
financial  management  education  and  training,  a  comprehensive  list 
of  these  providers  needed  to  be  verified. 

The  principal  sources  for  determining  the  agencies  and  departments 
to  be  surveyed  were: 

(1)  Formal  Schools  Directory.  Third  Edition  [Ref.  8:]  dated 
January,  1992.  This  directory  was  prepared  by  the  DoD  Training  and 
Performance  Data  Center  (TPDC)  which  is  located  in  Orlando, 
Florida.  The  directory  lists  all  schools  and  training  locations 
operated  by  the  Military  Services  and  DoD  agencies,  including  those 
identified  as  financial  management.  Five  hundred  eighty  nine 
training  organizations  are  Identified  in  this  directory. 

(2)  Financial  Management  Data  System  [Ref.  9:]  also 
developed  by  TPDC.  The  Financial  Management  Data  System  is  an 
automated  data  collection  system  which  is  designed  to  facilitate 
the  collection  of  descriptive  and  resource  data  for  DoD  financial 
management  training  and  education  courses.  TPDC  provided  this 
researcher  with  the  latest  edition  of  the  data  base  which  was 
updated  through  January,  1992. 

The  financial  management  data  base  was  cross-referenced  to  the 
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Formal  Schools  Directory  [Ref.  8:]  to  ensure  that  the  latest 
available  data  on  providers  of  financial  management  education  and 
training  was  utilized. 

A  final  check  was  performed  to  ensure  that  all  providers  of 
financial  management  education  and  training  were  included.  This 
included  correspondence  and  communication  with  experts  both  at  the 
Naval  Postgraduate  School  and  with  field  personnel  in  the  various 
military  departments.  The  final  product  or  listing  of  providers 
was  then  prepared. 

DEVELOPMENT  OF  THE  SURVEY  INSTRUMENT 

The  development  of  the  survey  used  to  gather  data  for  this 
research  began  with  an  extensive  review  of  the  literature  on  survey 
design.  [See  references  10  through  21]  Since  the  full  spectrum  of 
assessment  techniques  employed  by  DoD  financial  management 
education  and  training  programs  was  not  known,  the  basic 
methodological  approach  employed  in  the  development  of  the 
questionnaire  was  the  "open”  response  format.  This  format  allows 
the  recipient  to  reply  to  the  questions  in  a  manner  best  suited  to 
the  institutional  setting  of  the  respondent.  [Ref.  11:  p.  54] 
Additionally,  filtering  questions  were  developed  to  facilitate  the 
response  to  questions  that  did  not  apply  to  certain  recipients  but 
did  apply  in  general.  Finally,  general  purpose  or  information 
questions  were  included  in  the  question  data  base. 

A  second  methodological  issue  that  was  addressed  was  who 
should  be  contacted  in  the  survey  process.  Mr.  Peter  Ewell,  Senior 
Associate  with  the  National  Center  for  Higher  Education  Management 


2 


Systems  (NCHEMS)  noted  that  quality  assessment  of  education  could 
be  conducted  in  two  principal  ways.  These  methods  are: 

(1)  Collection  from  the  agencies  and  departments  themselves 
providing  an  Internal  perspective,  and 

(2)  Collection  of  information  from  the  clients  served  by  the 
agencies  and  departments. 

The  clients  are  either  the  students  themselves  or  the 
sponsoring  agencies  which  subsequently  received  the  students.  This 
method  of  data  gathering  is  referred  to  as  external.  [Ref.  22] 
Due  to  the  limited  time  frame  and  the  prospect  of  uncertain 
response  from  the  external  sources,  the  internal  method  was 
selected. 

The  third  methodological  issue  addressed  was  the  data 
collection  method  to  be  employed  in  the  survey.  Given  the  time 
constraints  as  well  as  the  perceived  length  of  the  questionnaire  on 
financial  management  quality  assessment  techniques,  a  combination 
of  telephone  and  mail  survey  techniques  were  selected.  In  addition 
to  the  probability  of  gaining  higher  reliability  of  the  responses, 
this  method  allowed  the  survey  respondent  to  include  additionally 
materials  to  the  basic  survey  document. 

Initial  Development  of  the  Survey  Questions 

Once  the  decision  had  been  made  by  DoD  Comptroller  staff  to 
perform  a  quality  assessment  of  financial  management  education  and 
training  courses  and  programs,  the  initial  development  of  the 
survey  instrument  was  undertaken.  In  this  phase  of  development, 
two  steps  were  followed.  The  first  step  was  to  review  research 
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literature  on  survey  methodology.  (Detailed  in  the  previous 
section) . 

The  second  step  was  to  contact  professionals  in  the  field  of 
educational  assessment  to  provide  additional  guidance  on  the  survey 
design. 

One  of  the  more  significant  contributions  at  this  stage  was 
provided  by  Peter  T.  Ewell,  Senior  Associate  at  the  National  Center 
for  Higher  Education  Management  Systems  located  in  Boulder, 
Colorado.  In  his  response  to  a  request  for  assistance  in 
developing  a  comprehensive  survey,  Mr.  Ewell  recommended  the 
following  procedures  be  assessed: 


•  The  nature  of  the  program  with  respect  to  target  audience 
(e.g.,  civilian/military,  rank,  geographical  region)  and  its 
intensity/duration  (e.g.,  one  day,  two-week  on  site,  one  year 
part-time,  correspondence  course) . 

•  The  learning  objectives  of  the  program,  framed  if  possible  in 
terms  of  "expected  outcomes"  of  instruction  i.e.,  particular 
elements  of  knowledge  or  skill  that  a  student  will  exhibit  at 
the  programs  conclusion. 

•  The  particular  methods  or  modes  of  Instruction  used  in 
delivering  the  program  (e.g.,  hands-on  training,  simulation, 
classroom  work,  independent  study,  etc.),  and  in  particular 
the  ways  in  which  students'  competency  is  tested  at  its 
conclusion  (e.g.,  paper-and-pencil  exams,  problems,  rated 
demonstrations  of  hands-on  performance,  etc.). 

•  How  the  effectiveness  of  the  program  in  attaining  its  training 
objectives  has  been  evaluated  in  the  past;  if  possible,  units 
should  be  requested  to  attach  copies  of  any  studies  or  data  on 
effectiveness  that  they  routinely  collect  to  monitor  and 
improve  their  own  performance.  [Ref.  22] 


Field  Validation  of  the  Survey  Questions 

Once  the  initial  set  of  survey  questions  had  been  compiled, 
the  "validity"  and  "relevance"  of  these  questions  needed  to  be 
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tested.  To  accomplish  this  testing,  selected  agencies  and 
departments  offering  financial  management  education  and  training 
programs  were  contacted  through  a  telephone  survey.  The 
methodology  employed  in  this  telephone  survey  was  "blind"  response, 
i.e.,  the  respondent  was  asked  general  questions  on  quality 
assessment  and  then  asked  what  c[uestions  they  would  consider 
relevant  in  developing  a  survey  instrument  for  their  facility.  Ten 
of  the  31  institutions  previously  identified  as  providers  of 
financial  management  education  and  training  were  contacted  in  this 
process.  The  providers  were  selected  on  the  basis  of  size,  service 
branch  and  referral  from  previously  contacted  providers.  The 
various  questions,  suggestions  and  pertinent  information  collected 
during  the  telephone  survey  were  added  to  the  data  base  of 
questions  prepared  prior  to  the  telephone  survey.  Appendix  A  lists 
those  agencies  and  departments  contacted  during  this  phase  of 
survey  preparation. 

The  Initial  survey  Instrument 

Once  the  agencies  and  departments  were  selected  for  survey  and 
the  survey  questions  to  be  asked  had  been  collected  and  developed, 
the  next  phase  was  to  organize  the  survey  questions  into  logical 
units.  Specifically,  the  questions  that  had  been  obtained  through 
the  research  noted  previously  had  to  be  organized  into  a  cohesive 
survey  document.  The  general  categories  and  principle  questions  or 
information  requested  is  listed  below: 

1.  General  organizational  and  program  information  including: 
respondent,  point  of  contact,  target  audience,  number  of 
financial  management  courses  and  programs  offered,  length 
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of  the  program,  numbers  of  times  offered,  annual  enrollment 
and  average  class  size  among  others. 

2.  Program  requirements  and  development;  How  is  the  need 
determined  for  new  courses?  How  is  the  need  to  update  or 
modify  courses  determined?  Is  course  development  guided  by 
higher  authority?  and  Does  the  agency  or  department  have 
staff  dedicated  to  program  development? 

3.  Instruction;  What  policies  are  used  to  hire 

faculty/ instructors?  What  methods  are  use  to  evaluate 

instruction  and  instructors? 

4.  Ongoing  evaluation;  What  methods  are  used  to  determine  the 
need  for  the  current  program?  What  methods  are  used  to 
determine  student  competency?  Are  formal  methods  used  for 
student  course  evaluation?  and  What  percentage  of  students 
successfully  complete  the  program? 

5.  Program  assessment ;  Is  the  performance  of  graduates 
tracked?  What  formal  methods  are  employed  to  solicit 
feedback  from  former  students  and  clients  of  those 
students? 


Once  the  questions  were  organized  as  noted  above,  the  complete 
survey  questionnaire  was  ready  for  initial  pilot  testing.  The 
survey  was  given  to  CDR  Glenn  Eberling  who  taught  the  Navy 
Practical  Control lership  Course  at  the  Naval  Postgraduate  School  in 
Monterey,  California  and  to  Mr.  Wade  Cliendienst  who  was  the  Chief 
of  Training  and  Professional  Development  for  the  US  Army  Audit 
Agency  located  in  Alexandria,  Virginia. 

Both  recipients  reviewed  the  survey  and  offered  several 
additions  to  the  proposed  survey  instrument.  These  additions 
focused  on  the  methods  of  quality  assessment  and  the  organization 
of  the  survey.  Both  pilot  recipients  also  suggested  clarification 
in  the  wording  of  the  questions  and  of  the  explanatory  paragraphs 
included  in  the  survey. 

A  draft  of  the  survey  was  reviewed  by  DoD  Comptroller  staff 
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and  a  cover  letter  from  DoD  Comptroller  Sean  O'Keefe  was  provided 
to  introduce  the  survey.  Sponsorship  of  the  survey  by  the 
Comptroller  was  essential  to  ensure  support  from  the  agencies  and 
departments  surveyed.  The  cover  letter  from  the  DoD  Comptroller  is 
included  at  the  end  of  the  Appendix  B.  Of  the  31  surveys  sent  to 
providers  of  financial  management  education  and  training,  21  were 
returned.  The  21  responses  were  received  prior  to  April  30,1992, 
the  cut-off  date  for  inclusion  in  the  research  project.  This 
represents  a  response  rate  of  68  percent.  Surveys  returned  by  the 
Center  for  Army  Leadership  and  the  Army's  Judge  Advocate  General's 
School  indicated  that  neither  agency  offered  financial  management 
education  and  training. 

The  response  rate  based  on  a  revised  total  of  29  providers 
(the  original  thirty-one  less  the  two  mentioned  above)  with  19 
responses  is  67  percent^. 

Another  measurement  of  the  response  is  to  calculate  the  number 
of  financial  management  programs  managed  by  agencies  and 
departments  who  responded  to  the  survey.  By  this  reckoning,  the 


^  An  effort  was  made  to  obtain  a  100  percent  response  rate  to 
the  survey.  Three  non-responding  agencies  were  specifically 
identified  for  intensive  follow-up  due  to  either  their  size  or 
number  of  course  offerings.  They  were  the  DLA  Finance  Training 
Section,  the  Defense  Systems  Management  College  and  the  Army  Audit 
Agency.  As  a  result  of  these  efforts,  surveys  from  the  Defense 
Systems  Management  College  and  DLA  Finance  Training  Section  were 
received  after  the  primary  research  on  the  thesis  had  been 
completed.  The  responses  from  these  two  agencies  conformed  to  the 
results  obtained  from  the  nineteen  survey  respondents  noted  in  this 
section.  Also,  Don  Cress  of  the  Army  Audit  Agency  reported  that  he 
Intended  to  complete  and  return  the  survey.  A  revised  response 
rate  based  on  22  responses  is  76  percent.  The  other  non-responding 
agencies  were  contacted  if  the  point  of  contact  was  known,  but  no 
other  follow-up  was  conducted. 
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total  number  of  financial  management  programs  offered  totals  181. 
Respondents  to  the  survey  manage  134  of  the  181  programs.  The 
response  rate  based  on  this  measure  is  79  percent^. 

Tables  V(A)  and  V(B)  detail  the  survey  questions  asked  of 
respondents  and  the  actual  number  of  responses  provided  to  each 
question.  The  variability  in  responses  to  the  questions  is  based 
on  (1)  the  non-applicability  of  certain  question  to  the  various 
respondents  and  (2)  the  failure  of  certain  respondents  to  answer 
specific  questions.  The  variability  in  response  rate  effects  the 
discussion  which  follows. 


^  Including  DLA,  Defense  Systems  Management  College  and  the 
Army  Audit  Agency,  the  response  rate  is  87  percent;  158  of  181 
programs . 
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SURVEY  QUESTION 


TARGET  AUDIENCE 


PERCENTAGE  OF  CLASS  ATTENDEES 


GRADES/RANKS  OF  ATTENDEES 


NUMBER  OF  COURSES 


LENGTH 


TIMES  OFFERED 


ANNUAL  ENROLIKENT 


AVERAGE  CLASS  SIZE 


COLLEGE  CREDITS 


DEGREE  OR  CERTIFICATE  OFFERED 


HOW  DETERMINE  NEED  FOR  NEW  COURSES 


HOW  DETERMINE  NEED  TO  UPDATE  COURSES 


SPONSORS 


COURSE  DEVELOPMENT  GUIDED 


TYPES  OF  GUIDANCE  RECEIVED 


HOW  OFTEN  IS  GUIDANCE  RECEIVED 


STAFF  DEDICATED  TO  PROGRAM  DEVELOPMENT 


PERCENTAGE  OF  MIL. /CIV.  INSTRUCTORS 


POLICIES/CRITERIA  TO  HIRE  MILITARY 


POLICIES/CRITERIA  TO  HIRE  CIVILIANS 


PROFESSIONAL  QUALIFICATION  STANDARDS 


CLASSROOM  OBSERVATION 


OTHER  METHODS  TO  EVALUATE  INSTRUCTION 


OTHER  METHODS  TO  EVALUATE  INSTRUCTORS 


MO.  OF  RESPONSES 
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25. 

PRODUCTIVITY  MEASURES 

19  1 

26. 

INSTITUTION  ACCREDITED 

19  1 

27. 

INSTITUTION  REVIEWED 

19  1 

28. 

NATURE  OF  REVIEW 

9 

29. 

METHODS  OF  INSTRUCTION 

19 

30. 

PERCENTAGE  OF  INSTRUCTION  BY  METHOD 

11 

31. 

TYPES  OF  COURSE  READING  MATERIAL 

16 

1 

LIBRARY  SUPPORT 

18 

1 
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6 

1 

IMPROVEMENTS  PLANNED  IN  LIBRARY 

17 

1 

COMPUTER  SUPPORT 

19 

36. 

COMPUTER  EXERCISES 

18 

37. 

PROGRAMMATIC  SUPPORT 

19 

38. 

METHODS  TO  DETERMINE  NEED  FOR  PROGRAM 

18 

39. 

METHODS  TO  ASSESS  VALIDITY,  RELEVANCE 

19 

40. 

METHODS  TO  DETERMINE  COMPETENCE 

16 

41. 

FORMAL  METHOD  OF  STUDENT  EVALUATION 

19 

42. 

OTHER  METHODS  FOR  STUDENT  EVALUATION 

19 

43. 

PASS  STANDARDIZED  EXAMS  OR  TESTS 

19 

44. 
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11 
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19 

46. 

FEEDBACK  FROM  FORMER  STUDENTS 

19 

LL 

MOST  IMPORTANT  METHODS  OF  QUALITY 

19 
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APPENDIX  B: 


DEPARTMENT  OF  DEFENSE  FINANCIAL  MANAGEMENT  EDUCATION 

AND  TRAINING  QUALITY  ASSESSMENT  SURVEY 

PURPOSE ;  This  survey  questionnaire  is  designed  to  determine  how 
DoD  financial  management  education  and  training  programs  assess  the 
quality  of  their  programs.  Quality  in  the  context  of  this  survey 
means  providing  accurate,  valid,  comprehensive  and  up-to-date 
information  to  meet  the  needs  of  your  customers  or  clients. 
Quality  begins  with  the  determination  of  the  need  for  the  program 
and  continues  through  the  monitoring  of  the  performance  of 
graduates.  Quality  consists  of  providing  the  correct  instruction 
for  the  appropriate  target  population. 

The  following  series  of  questions  is  designed  to  allow  you  to 
provide  information  on  quality  assessment  techniques  employed  by 
your  organization.  We  encourage  you  to  answer  the  questions 
completely  and  add  additional  information  wherever  appropriate. 
Please  attach  additional  pages  and  materials  as  necessary  to 
provide  complete  answers. 


ORGANIZATIONAL  INFORMATION: 
Respondent's  Name: 


Position: 

Agency/Component  Name: 
Address : 


Phone  Number: 


(DSN/Autovon) 


Please  provide  the  title  or  name  of  the  instructional  program 
offered  at  your  institution: 


GENERAL  PROGRAM  INFORMATION: 

1.  What  is  the  target  audience  for  your  program? 

(CIVILIAN)  (MILITARY)  (BOTH) 

2.  What  are  the  approximate  percentages  of  class  attendees? 

CIVILIAN: _  % 

MILITARY: _  % 

3.  What  are  the  ranges  of  grades  or  ranks  of  your  attendees? 

CIVILIAN: _ 

(GS  or  GM  -  to  GS  or  GM  -  ;  SES  Level) 

MILITARY : _ 

4.  How  many  separate  courses  in  financial  management  are 
offered  within  your  program? 


5.  What  is  the  length  of  your  program  in  days,  weeks  or 

months?  _ 

6.  How  many  times  per  year  is  it  offered?  _ 

7.  What  is  the  annual  enrollment  in  your  program? 


8.  What  is  the  average  (mean)  class  size? 


9.  If  your  program  provides  course  credit  for  college,  how  many 
credits  are  offered  in  the  total  program? 


10.  Do  students  completing  the  program  receive  a  degree  or 
certificate? 

(YES)  (NO) 

If  yes,  what  is  the  title  of  this  award? 


The  rest  of  the  (questionnaire  is  organized  into  the  following 
parts:  (1)  Program  requirements  and  development  (2)  Instruction  (3) 
Ongoing  evaluation  and  (4)  Program  assessment.  Some  of  the 
questions  require  a  simple  YES/NO  response,  other  ask  for  short 
responses,  helpful  comments  and  recommendations.  We  ask  you  to 
carefully  complete  this  survey  so  that  we  can  develop  a 
comprehensive  data  base  of  quality  assessment  measures  employed  in 
financial  management  education  and  training. 


PROGRAM  REQUIREMENTS  AND  DEVELOPMENT; 

11.  How  do  you  determine  need  for  new  courses  or  instruction? 


12.  How  do  you  determine  the  need  to  update  or  modify  the 
instruction  you  currently  offer? 


13.  Do  you  have  one  or  more  sponsors  or  clients  that  directly 
influence  your  program  of  Instruction? 


(YES) 


(NO) 


If  yes,  who  is/are  the  sponsor (s)? 


14.  Is  course  development  guided  or  directed  by  higher 
authority? 

(YES)  (NO) 

If  so, 

who? _ 

15.  What  types  of  guidance  do  you  receive? 


16.  How  often  do  you  receive  such  guidance? 


17.  Do  you  have  staff  dedicated  to  program  development? 
(YES)  (NO) 

If  yes,  how  many? _ 


INSTRUCTION: 

18.  What  are  the  number  of  and  percentages  of  civilian  and/or 
military  instructors  in  your  program? 

CIVILIAN;  Number _  Percentage _ 

MILITARY:  Number _  Percentage _ 


19.  What  policies  and  criteria  are  used  to  assign  or  hire 

military  faculty/ instructors  for  your  program?  Please  attach 
any  material  or  guidance  that  describes  these  policies  or 
criteria  to  your  survey  response. 


What  policies  and  criteria  are  used  to  appoint  or  hire 
civilian  faculty/ instructors  for  your  program?  Please  attach 
any  material  or  guidance  that  describes  these  policies  and 
criteria  to  your  survey  response. 


21.  Are  professional  qualifications  standards  required  for 
faculty/ instructors  (e.g.,  degrees,  prior  experience, 
completion  of  an  instructor  training  program) ? 

(YES)  (NO) 

If  yes,  please  describe.  Attach  written  guidance  or 
material  that  describes  such  qualifications  to  your  survey 
response. 


22.  Do  you  use  classroom  observation  to  evaluate  instruction? 
(YES)  (NO) 

If  yes,  who  pe^-forms  this  obseirvation? 


How  often  is  it  performed? 


23.  Are  other  methods  used  to  evaluate  classroom  instruction? 
(YES)  (NO) 


If  yes,  please  describe: 


24.  Are  other  methods  used  to  evaluate  instructors? 
(YES)  (NO) 

If  yes,  please  describe: 


25.  Do  you  employ  productivity  measures  to  evaluate  instructors? 
(YES)  (NO) 

If  so,  please  describe.  Attach  any  materials  that 
describe  this  process  to  your  survey  response. 


26.  Is  your  institution  accredited? 

(YES)  (NO) 

If  yes,  by  whom?  _ 

How  often  is  the  accreditation  review  performed? 


27.  Is  your  institution  reviewed  by  other  agencies? 
(YES)  (NO) 

If  yes,  by  whom  and  how  often? 


28.  What  is  the  nature  and  extent  of  this  review?  Please  attach 
any  materials  that  describes  this  process  to  your  survey 
response . 


29.  What  methods  or  modes  of  instruction  are  utilized  in  your 

program  (e.g.,  lecture,  discussion,  simulation,  independent 
study) ? 


30.  What  percentage  of  instruction  is  delivered  in  each  mode  you 
identified? 


31.  What  types  of  course  reading  materials  are  used  in  your 

program  (e,g.,  textbooks,  published  articles,  DoD  official 
documents,  self-generated  materials)? 


32.  Is  your  program  supported  by  a  library  containing  extensive 
DoD  financial  management  related  materials? 

(YES)  (NO) 

33.  How  would  you  like  to  improve  library  support  for  your 
program? 


34.  Are  there  improvements  in  your  library  services  planned? 


(YES)  (NO) 

If  yes,  please  explain  these  improvements  and  indicate 
when  they  will  occur. 


35.  Is  your  program  supported  by  a  computer  center  or 
laboratory? 

(YES)  (NO) 

If  no,  what  computer  resources  are  employed  at  your 
institution? 


36.  Does  your  instructional  program  include  computer  exercises 
or  assignments? 

(YES)  (NO) 

If  yes,  please  describe  how  computers  are  utilized. 


37.  Is  your  program  well  supported  with  respect  to 

administrative  staff,  funding  for  guest  speakers,  etc.? 

(YES)  (NO) 

What  Improvements  in  support  would  be  desirable? 


ONGOING  EVALUATION; 

38.  What  methods  are  used  to  determine  the  need  for  your  current 
program? 


39.  What  methods  are  used  to  determine  the  validity,  accuracy, 
relevance  and  timeliness  of  your  current  program? 


40.  What  methods  are  employed  to  determine  student  competence 
during  and  upon  program  completion  (e.g.,  passing 
standardized  tests,  written  course  work,  observation  of 
perfozrmance  on  the  job)? 


41.  Do  you  have  a  formal  method  of  student  course  evaluation? 
(YES)  (NO) 

If  yes,  describe  this  process  and  provide  the  form  used 
to  gather  student  evaluations  with  your  response. 


42.  Are  other  methods  used  to  permit  students  to  evaluate  your 
instructional  program? 


(YES)  (NO) 

If  yes,  describe  these  methods: 


43.  Are  your  students  required  to  pass  any  standardized  OoD  or 
national  examinations  (e.g.,  CPA,  CMA)? 


(YES) 


(NO) 


If  yes,  please 

identify: _ 

44.  What  percentage  of  students  entering  your  program 

successfully  complete  it?  _ % 

PROGRAM  ASSESSMENT; 

45.  Do  you  track  the  performance  of  your  graduates? 

(YES)  (NO) 

If  yes,  how  is  this  done? 


46.  Are  formal  methods  employed  to  solicit  feedback  from  former 
students  and  clients  about  the  utility  of  your  program? 

(YES)  (NO) 

If  yes,  please  explain  how  this  is  done  and  provide 
examples  of  such  feedback. 


47.  In  your  opinion,  what  are  the  most  important  methods  you 
employ  to  ensure  the  quality  of  your  program? 


48.  Is  there  any  other  Information  that  you  would  offet  on 
quality  assessment  at  your  institution  that  we  did  not 
request?  Please  provide  comments  as  appropriate. 


APPENDIX  C 


DoD  FINANCIAL  MANAGEMENT  AGENCIES  AND  DEPARTMENTS 
RESPONDING  TO  THE  SURVEY 

ARMY  FINANCIAL  MANAGEMENT  SCHOOLS; 


US  Army  Finance  School 

Fort  Benjamin  Harrison,  IN  46216-5640 

US  Army  Management  Engineering  College 
Rock  Island  Arsenal,  IL  61229-7040 

Syracuse  University 
Army  Programs  Office 
310  School  of  Management 
Syracuse,  NY  13244-2130 

US  Army  Engineer  Division 
Directorate  of  CE  Training  Management 
Huntsville,  AL  35807-4301 

US  Army  Logistics  Management  College 
Fort  Lee,  VA  23801-6056 

*  The  Judge  Advocate  General's  School 
Contract  Law  Division 
Charlottesville,  VA  22903-1781 

*  Center  for  Army  Leadership 
Fort  Leavenworth,  KS  66027 


AIR  FORCE  FINANCIAL  MANAGEMENT  SCHOOLS: 


3750TH  Technical  Training  Group 
Sheppard  Air  Force  Base,  TX  76311-5434 

Air  Force  Institute  of  Technology 
Nrlght-Patterson  Air  Force  Base,  OH  45433-6583 

Air  University 

Professional  Military  Comptroller  School 


Maxwell  Air  Force  Base,  AL  36112-5712 


NAVY  FINANCIAL  MANAGEMENT  SCHOOLS; 


Naval  Postgraduate  School 
Department  of  Administrative  Sciences 
Monterey,  CA.  93943-5022 

American  University 
Washington,  DC  20016-8070 

Troy  State  University 
P.  O.  Box  1032 
Fort  Meyer,  VA  22211 

Navy  Comptroller 

Program  Management  Office 

Naval  Air  Station  Pensacola 

Building  625D 

Pensacola,  FL  32508-5175 

Fleet  Training  Center 
Norfolk,  VA  23511 

Civil  Engineer  Corps  Officers  School 
Port  Hueneme,  CA  93043-5002 

Naval  School  of  Health  Sciences 
Bethesda,  MD  20814-5033 

Fleet  Aviation  Specialized  operational  Training  Group 
Norfolk,  VA  23511 

Navy  School  of  Manpower  Management 
Norfolk,  VA  23511 

Naval  Military  Personnel  Command 
Navy  MWR  Recreation  Training  Unit 
Patuxent  River,  MD  20670 


MARINE  CORPS  FINANCIAL  MANAGEMENT  SCHOOLS; 

Financial  Management  School 
Marine  Corps  Service  Support  Schools 
Camp  Lejeune,  NC  28542-5050 


NOTE:  An  asterisks  preceding  the  name  indicates  that  a 

response  was  received.  However,  the  response  was  that  the 
agency  or  departnient  did  not  perfom  financial  inanageinent 
education  and  training. 
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MEMORAMDOM  FOR  ASSISTANT  SECRETARY  OP  TEE  ARMY  (FINANCIAL 

MANAGEMENT) 

ASSISTANT  SECRETARY  OF  TEE  NAVY  (FINANCIAL 
MANAGEMENT) 

ASSISTANT  SECRETARY  OF  TEE  AIR  FORCE  (FINANCIAL 
MANAGEMENT  AND  COMPTROLLER) 

DIRECTOR  OF  THE  JOINT  STAFF 

SUBJECT:  Financial  Management  Review 


Within  the  next  couple  o£  months  you  may  be  contacted  by 
Professor  Larry  R.  Jones  from  the  Naval  Postgraduate  School 
requesting  interviews  with  you  and  your  staff  concerning  the 
relationship  between  the  Defense  Management  Report  (DMR)  and  DOD 
financial  management. 

Among  the  benefits  that  I  expect  from  Professor  Jones' 
efforts  are: 

•  an  academic  review  of  the  DMR  process  on  financial 
management  within  DOD, 

•  an  evaluation  of  financial  management  changes 
undertaken  within  the  OMR  process / 

•  some  insights  into  the  role  financial  management 
education  resources  should  play  into  the  DMR  process r 
and 

•  some  insights  into  the  future  course  of  financial 
management  with  respect  to  the  Defense  Management 
Report. 

Your  cooperation  with  Professor  Jones  will  be  beneficial  to 
us  all  and  greatly  appreciated. 
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